[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:48 dist-test-slave-m1qq krb5kdc[19128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:48 dist-test-slave-m1qq krb5kdc[19128](info): set up 2 sockets
Sep 24 20:19:48 dist-test-slave-m1qq krb5kdc[19128](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 20:19:49.288405 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:19:49 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:19:49.423672 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34883
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34883
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:49.537696 19141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:49.538060 19141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:49.538153 19141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:49.542465 19141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:49.542591 19141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:49.542634 19141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:49.542675 19141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:49.542711 19141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:49.548255 19141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:49.549448 19141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:49.550542 19141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:49.560326 19146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:49.560340 19149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:49.560346 19147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:49.560406 19141 server_base.cc:1034] running on GCE node
I20260924 20:19:49.561120 19141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:49.562258 19141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:49.563421 19141 hybrid_clock.cc:648] HybridClock initialized: now 1790281189563390 us; error 43 us; skew 500 ppm
Sep 24 20:19:49 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281189, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:49.569888 19141 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:19:49.570757 19141 webserver.cc:460] Webserver started at http://127.18.172.126:44577/ using document root <none> and password file <none>
I20260924 20:19:49.571365 19141 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:49.571441 19141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:49.571666 19141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:49.573855 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "3f77a21148d2480aaffd24cec9209afb"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "8ed5f7b10458002ae5571025a68bea8c"
server_key_iv: "8cfb353f13d3ba4641df85e7f0311bd4"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.574456 19141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "3f77a21148d2480aaffd24cec9209afb"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "8ed5f7b10458002ae5571025a68bea8c"
server_key_iv: "8cfb353f13d3ba4641df85e7f0311bd4"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.578328 19141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:19:49.580894 19156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:49.582157 19141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:19:49.582343 19141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "3f77a21148d2480aaffd24cec9209afb"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "8ed5f7b10458002ae5571025a68bea8c"
server_key_iv: "8cfb353f13d3ba4641df85e7f0311bd4"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.582463 19141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:49.632006 19141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:49.635478 19141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:49.635703 19141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:49.643204 19141 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:34883
I20260924 20:19:49.643304 19208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:34883 every 8 connection(s)
I20260924 20:19:49.644277 19141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:49.647464 19209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:49.651154 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19141
I20260924 20:19:49.651321 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:19:49.652809 19121 external_mini_cluster.cc:1428] Setting key a4ffdd9b2e722a00cf7d3a0f8ca1c0a6
I20260924 20:19:49.653316 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb: Bootstrap starting.
I20260924 20:19:49.656013 19209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:49.656878 19209 log.cc:824] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:49.658843 19209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb: No bootstrap required, opened a new log
I20260924 20:19:49.663419 19209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } }
I20260924 20:19:49.663664 19209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:49.663749 19209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f77a21148d2480aaffd24cec9209afb, State: Initialized, Role: FOLLOWER
I20260924 20:19:49.664171 19209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } }
I20260924 20:19:49.664322 19209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:49.664410 19209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:49.664528 19209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:49.665597 19209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } }
I20260924 20:19:49.665953 19209 leader_election.cc:302] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f77a21148d2480aaffd24cec9209afb; no voters:
I20260924 20:19:49.666257 19209 leader_election.cc:288] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:49.666381 19214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:49.666620 19214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [term 1 LEADER]: Becoming Leader. State: Replica: 3f77a21148d2480aaffd24cec9209afb, State: Running, Role: LEADER
I20260924 20:19:49.667050 19214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } }
I20260924 20:19:49.667528 19209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:49.668587 19215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f77a21148d2480aaffd24cec9209afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } } }
I20260924 20:19:49.668728 19215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:49.669040 19216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f77a21148d2480aaffd24cec9209afb. Latest consensus state: current_term: 1 leader_uuid: "3f77a21148d2480aaffd24cec9209afb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f77a21148d2480aaffd24cec9209afb" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34883 } } }
I20260924 20:19:49.669131 19216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:49.669462 19223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:49.672313 19223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 20:19:49 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281189, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:49.678191 19223 catalog_manager.cc:1347] Generated new cluster ID: 361d8924b39f4bb3af6dc15eca56d2ef
I20260924 20:19:49.678299 19223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:49.681048 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:49.655988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.18.172.126:34883)
0924 20:19:49.656512 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:49.656529 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:49.657852 (+ 1323us) server_negotiation.cc:407] Connection header received
0924 20:19:49.661816 (+ 3964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:49.661839 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:49.662274 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:49.662692 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:49.665145 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.666613 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.668345 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.669146 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.669334 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:49.674520 (+ 5186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:49.674526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:49.674539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:49.674570 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:49.676650 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:49.677183 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:49.677194 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:49.677199 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:49.677316 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:49.677692 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:49.677694 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:49.677695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:49.678052 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:49.679814 (+ 1762us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:49.679951 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 20:19:49.680212 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":369,"thread_start_us":163,"threads_started":1}
I20260924 20:19:49.695806 19223 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:49.696753 19223 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:49.705808 19223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f77a21148d2480aaffd24cec9209afb: Generated new TSK 0
I20260924 20:19:49.706532 19223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:19:49.748052 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:49.879356 19235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:49.879614 19235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:49.879662 19235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:49.883664 19235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:49.883723 19235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:49.883782 19235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:19:49.889159 19235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:49.890157 19235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:49.891202 19235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:49.901260 19243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:49.901336 19235 server_base.cc:1034] running on GCE node
W20260924 20:19:49.901237 19240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:49.901237 19241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:49.901918 19235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:49.902721 19235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:49.903964 19235 hybrid_clock.cc:648] HybridClock initialized: now 1790281189903951 us; error 26 us; skew 500 ppm
Sep 24 20:19:49 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281189, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:49.906910 19235 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:19:49.907807 19235 webserver.cc:460] Webserver started at http://127.18.172.65:36083/ using document root <none> and password file <none>
I20260924 20:19:49.908416 19235 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:49.908511 19235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:49.908774 19235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:49.910804 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "c1adfbc92ccf4695a0b6cd7bc9f4ddaa"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "922bb8dc165b34740ec57d299f817664"
server_key_iv: "7ff1a655833f3fd0f39754308dc15ede"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.911439 19235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "c1adfbc92ccf4695a0b6cd7bc9f4ddaa"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "922bb8dc165b34740ec57d299f817664"
server_key_iv: "7ff1a655833f3fd0f39754308dc15ede"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.915302 19235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:19:49.917819 19250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:49.919183 19235 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:19:49.919314 19235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "c1adfbc92ccf4695a0b6cd7bc9f4ddaa"
format_stamp: "Formatted at 2026-09-24 20:19:49 on dist-test-slave-m1qq"
server_key: "922bb8dc165b34740ec57d299f817664"
server_key_iv: "7ff1a655833f3fd0f39754308dc15ede"
server_key_version: "encryptionkey@0"
I20260924 20:19:49.919433 19235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:49.933328 19235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:49.937073 19235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:49.937301 19235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:49.937959 19235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:49.938858 19235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:49.938941 19235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:49.939045 19235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:49.939087 19235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:49.948856 19235 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:42265
I20260924 20:19:49.948892 19363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:42265 every 8 connection(s)
I20260924 20:19:49.949820 19235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:19:49.955667 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19235
I20260924 20:19:49.955767 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:19:49.956074 19121 external_mini_cluster.cc:1428] Setting key b80192f63c711e5e24ef5703b5ab5c4e
Sep 24 20:19:49 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281189, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:49.966061 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:49.951735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46535 (local address 127.18.172.126:34883)
0924 20:19:49.951893 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:49.951898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:49.953008 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 20:19:49.953941 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:49.953943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:49.953989 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:49.954065 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:49.955508 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.956157 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.957827 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.958042 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.958121 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:49.961517 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:49.961524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:49.961530 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:49.961565 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:49.963350 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:49.964118 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:49.964120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:49.964122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:49.964198 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:49.964727 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:49.964729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:49.964730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:49.964914 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:49.964989 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:49.965832 (+ 843us) server_negotiation.cc:299] Negotiation successful
0924 20:19:49.965940 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:19:49.967476 19366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:49.952144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34883 (local address 127.18.172.65:46535)
0924 20:19:49.952850 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:49.952894 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:49.953748 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:49.954251 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:49.954259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:49.954669 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:49.955300 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:49.955317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.956302 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.956307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:49.957697 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:49.957710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:49.958215 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:49.958223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:49.958480 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:49.959233 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:49.959261 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:49.961320 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:49.963484 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:49.963494 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:49.963517 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:49.963994 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:49.964334 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:49.964340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:49.964345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:49.964588 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:49.965037 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:49.965045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:49.965422 (+ 377us) client_negotiation.cc:769] Sending connection context
0924 20:19:49.965802 (+ 380us) client_negotiation.cc:240] Negotiation successful
0924 20:19:49.966211 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":109,"threads_started":1}
I20260924 20:19:49.968922 19364 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34883
I20260924 20:19:49.969165 19364 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:49.969769 19364 heartbeater.cc:511] Master 127.18.172.126:34883 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:19:49.971455 19173 ts_manager.cc:194] Registered new tserver with Master: c1adfbc92ccf4695a0b6cd7bc9f4ddaa (127.18.172.65:42265)
I20260924 20:19:49.972975 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:46535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:19:49.991093 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:50.104049 19369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:50.104326 19369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:50.104372 19369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:50.108486 19369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:50.108547 19369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:50.108614 19369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:19:50.114030 19369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:50.115075 19369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:50.116047 19369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:50.126094 19374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:50.126173 19377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:50.126173 19375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:50.126266 19369 server_base.cc:1034] running on GCE node
I20260924 20:19:50.126693 19369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:50.127379 19369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:50.128538 19369 hybrid_clock.cc:648] HybridClock initialized: now 1790281190128502 us; error 50 us; skew 500 ppm
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:50.131372 19369 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:19:50.132228 19369 webserver.cc:460] Webserver started at http://127.18.172.66:37101/ using document root <none> and password file <none>
I20260924 20:19:50.132791 19369 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:50.132851 19369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:50.133066 19369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:50.135203 19369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "fb1997aba181484c9d26eb67b0b85fca"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "1a70c86db763afb6548b8ae19702b03a"
server_key_iv: "c60fea6e8474685732ce78ce2ed9e165"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.135771 19369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "fb1997aba181484c9d26eb67b0b85fca"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "1a70c86db763afb6548b8ae19702b03a"
server_key_iv: "c60fea6e8474685732ce78ce2ed9e165"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.139747 19369 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:19:50.142161 19384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.143611 19369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:50.143841 19369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "fb1997aba181484c9d26eb67b0b85fca"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "1a70c86db763afb6548b8ae19702b03a"
server_key_iv: "c60fea6e8474685732ce78ce2ed9e165"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.143975 19369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:50.162869 19369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:50.166079 19369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:50.166312 19369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:50.166961 19369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:50.168071 19369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:50.168120 19369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.168210 19369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:50.168251 19369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.178812 19369 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:43965
I20260924 20:19:50.178831 19497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:43965 every 8 connection(s)
I20260924 20:19:50.179993 19369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:19:50.187805 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19369
I20260924 20:19:50.187939 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:19:50.188205 19121 external_mini_cluster.cc:1428] Setting key 305ae2479d49859c7ea1a0cbbd289a10
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:50.195235 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.182032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57311 (local address 127.18.172.126:34883)
0924 20:19:50.182169 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:50.182173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:50.183695 (+ 1522us) server_negotiation.cc:407] Connection header received
0924 20:19:50.184598 (+ 903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:50.184601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:50.184655 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:50.184769 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:50.186110 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.186735 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.187915 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.188197 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.188321 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.191347 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:50.191353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:50.191361 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:50.191388 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:50.192969 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.193534 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.193540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.193542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.193636 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.194014 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.194017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.194019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.194221 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:50.194340 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:50.194876 (+ 536us) server_negotiation.cc:299] Negotiation successful
0924 20:19:50.195041 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:19:50.196089 19500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.182450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34883 (local address 127.18.172.66:57311)
0924 20:19:50.183531 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:50.183572 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:50.184393 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:50.184944 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:50.184950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:50.185441 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:50.185963 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:50.185977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.186860 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.186865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:50.187788 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:50.187797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.188367 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.188371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:50.188559 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.189162 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:50.189179 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:50.191169 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:50.193101 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:50.193107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:50.193126 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:50.193413 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:50.193747 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:50.193751 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:50.193753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:50.193901 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:50.194337 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:50.194344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:50.194605 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 20:19:50.194814 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 20:19:50.195202 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"thread_start_us":725,"threads_started":1}
I20260924 20:19:50.197584 19498 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34883
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:19:50.197919 19498 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:50.198489 19498 heartbeater.cc:511] Master 127.18.172.126:34883 requested a full tablet report, sending...
I20260924 20:19:50.199753 19173 ts_manager.cc:194] Registered new tserver with Master: fb1997aba181484c9d26eb67b0b85fca (127.18.172.66:43965)
I20260924 20:19:50.200496 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:57311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:19:50.219308 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:50.335156 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:50.335431 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:50.335476 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:50.340173 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:50.340245 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:50.340363 19503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:19:50.345969 19503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34883
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:50.347157 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:50.348244 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:50.358731 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:50.358630 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:50.358676 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:50.359117 19503 server_base.cc:1034] running on GCE node
I20260924 20:19:50.359488 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:50.360118 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:50.361301 19503 hybrid_clock.cc:648] HybridClock initialized: now 1790281190361204 us; error 109 us; skew 500 ppm
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:50.364596 19503 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:19:50.365747 19503 webserver.cc:460] Webserver started at http://127.18.172.67:40399/ using document root <none> and password file <none>
I20260924 20:19:50.366317 19503 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:50.366381 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:50.366556 19503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:50.369046 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "fe08d9e41bac4951b2eb57a8126f6f4a"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "3f16b58450a8fcd9136d1af799ff85dc"
server_key_iv: "07c2d1923575b93cdc0f42c5737d39a6"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.369727 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "fe08d9e41bac4951b2eb57a8126f6f4a"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "3f16b58450a8fcd9136d1af799ff85dc"
server_key_iv: "07c2d1923575b93cdc0f42c5737d39a6"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.374109 19503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:50.376861 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.378324 19503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:19:50.378434 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "fe08d9e41bac4951b2eb57a8126f6f4a"
format_stamp: "Formatted at 2026-09-24 20:19:50 on dist-test-slave-m1qq"
server_key: "3f16b58450a8fcd9136d1af799ff85dc"
server_key_iv: "07c2d1923575b93cdc0f42c5737d39a6"
server_key_version: "encryptionkey@0"
I20260924 20:19:50.378535 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:50.423030 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:50.426527 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:50.426692 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:50.427330 19503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:50.428122 19503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:50.428166 19503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.428208 19503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:50.428224 19503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:50.437117 19503 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42569
I20260924 20:19:50.437134 19631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42569 every 8 connection(s)
I20260924 20:19:50.438112 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:19:50.446863 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19503
I20260924 20:19:50.447033 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:19:50.447386 19121 external_mini_cluster.cc:1428] Setting key 153c9fae7a82d6f3394730ddb3d5aff6
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:50.453186 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.440357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:57861 (local address 127.18.172.126:34883)
0924 20:19:50.440509 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:50.440513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:50.441405 (+ 892us) server_negotiation.cc:407] Connection header received
0924 20:19:50.442728 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:50.442732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:50.442793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:50.442915 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:50.444202 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.444741 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.445963 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.446202 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.446330 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.449571 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:50.449575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:50.449578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:50.449607 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:50.451266 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.451801 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.451804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.451806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.451887 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.452203 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.452205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.452206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.452400 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:50.452477 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:50.452963 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 20:19:50.453066 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:19:50.454090 19634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.440761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34883 (local address 127.18.172.67:57861)
0924 20:19:50.441232 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:50.441287 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:50.442491 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:50.443116 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:50.443124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:50.443522 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:50.444044 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:50.444059 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.444880 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.444884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:50.445816 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:50.445830 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.446380 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.446384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:50.446573 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.447347 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:50.447363 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:50.449405 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:50.451376 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:50.451382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:50.451395 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:50.451657 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:50.451974 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:50.451977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:50.451978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:50.452116 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:50.452489 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:50.452494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:50.452745 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 20:19:50.452903 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 20:19:50.453163 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":124,"threads_started":1}
I20260924 20:19:50.455312 19632 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34883
I20260924 20:19:50.455574 19632 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:50.456127 19632 heartbeater.cc:511] Master 127.18.172.126:34883 requested a full tablet report, sending...
I20260924 20:19:50.457181 19173 ts_manager.cc:194] Registered new tserver with Master: fe08d9e41bac4951b2eb57a8126f6f4a (127.18.172.67:42569)
I20260924 20:19:50.457831 19173 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:57861
I20260924 20:19:50.462347 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:19:50.491223 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.480557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.18.172.65:42265)
0924 20:19:50.480981 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:50.480988 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:50.481013 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:19:50.481069 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:50.481075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:50.481281 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:50.481411 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:50.482089 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.483211 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.484118 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.484530 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.484623 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.487445 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:50.487456 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:50.487471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:50.487501 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:50.489346 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.489801 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.489808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.489809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.489909 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.490162 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.490164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.490165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.490435 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:50.490602 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:50.490783 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 20:19:50.490995 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":228,"threads_started":1}
W20260924 20:19:50.492369 19278 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36332
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:19:50.520583 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:50.510912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36348 (local address 127.18.172.65:42265)
0924 20:19:50.511117 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:50.511120 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:50.511204 (+ 84us) server_negotiation.cc:407] Connection header received
0924 20:19:50.511330 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:50.511333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:50.511377 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:50.511450 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:50.512474 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.513339 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.514642 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:50.514982 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:50.515119 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:50.517718 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:50.517722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:50.517724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:50.517751 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:50.519101 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.519515 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.519518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.519519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.519595 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:50.519850 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:50.519853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:50.519854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:50.520027 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:50.520164 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:50.520350 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 20:19:50.520450 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:19:50.521891 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19235
I20260924 20:19:50.528276 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19369
I20260924 20:19:50.535712 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19503
I20260924 20:19:50.541373 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19141
2026-09-24T20:19:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2399 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19646](info): set up 2 sockets
Sep 24 20:19:50 dist-test-slave-m1qq krb5kdc[19646](info): commencing operation
krb5kdc: starting...
W20260924 20:19:51.651437 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 20:19:51.868158 19121 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.217s user 0.001s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 20:19:52.128795 19121 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.260s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 20:19:52.309497 19121 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.181s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/test-user.keytab.
W20260924 20:19:52.956581 19121 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.647s user 0.001s sys 0.001s
Sep 24 20:19:52 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
W20260924 20:19:54.359177 19121 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.172.126: real 1.380s user 0.002s sys 0.001s
I20260924 20:19:54.359885 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43793
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:54.480795 19659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:54.481101 19659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:54.481143 19659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:54.485216 19659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:54.485275 19659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:54.485294 19659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:54.485311 19659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:54.485328 19659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:54.490645 19659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:54.491742 19659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:54.492758 19659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:54.502848 19664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:54.502858 19667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:54.502914 19659 server_base.cc:1034] running on GCE node
W20260924 20:19:54.502858 19665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:54.503818 19659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:54.505080 19659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:54.506296 19659 hybrid_clock.cc:648] HybridClock initialized: now 1790281194506269 us; error 39 us; skew 500 ppm
Sep 24 20:19:54 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281194, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:54.509248 19659 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:19:54.510372 19659 webserver.cc:460] Webserver started at http://127.18.172.126:41575/ using document root <none> and password file <none>
I20260924 20:19:54.510985 19659 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:54.511133 19659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:54.511379 19659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:54.513537 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "653d896e685f49308e5e5e1c5f23ae26"
format_stamp: "Formatted at 2026-09-24 20:19:54 on dist-test-slave-m1qq"
server_key: "2e456fc83bbf8600743578a42f0b9eb9"
server_key_iv: "a419c82897ba7351681faf87e8bc51c5"
server_key_version: "encryptionkey@0"
I20260924 20:19:54.514148 19659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "653d896e685f49308e5e5e1c5f23ae26"
format_stamp: "Formatted at 2026-09-24 20:19:54 on dist-test-slave-m1qq"
server_key: "2e456fc83bbf8600743578a42f0b9eb9"
server_key_iv: "a419c82897ba7351681faf87e8bc51c5"
server_key_version: "encryptionkey@0"
I20260924 20:19:54.518065 19659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 20:19:54.520788 19674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:54.522187 19659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:19:54.522365 19659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "653d896e685f49308e5e5e1c5f23ae26"
format_stamp: "Formatted at 2026-09-24 20:19:54 on dist-test-slave-m1qq"
server_key: "2e456fc83bbf8600743578a42f0b9eb9"
server_key_iv: "a419c82897ba7351681faf87e8bc51c5"
server_key_version: "encryptionkey@0"
I20260924 20:19:54.522490 19659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:54.538163 19659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:54.541936 19659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:54.542160 19659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:54.549654 19659 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43793
I20260924 20:19:54.549672 19726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43793 every 8 connection(s)
I20260924 20:19:54.550639 19659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:54.553742 19727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:54.557331 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19659
I20260924 20:19:54.557463 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:19:54.557741 19121 external_mini_cluster.cc:1428] Setting key 046f45e21195ac2a5e1f528e0521b493
I20260924 20:19:54.560133 19727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26: Bootstrap starting.
I20260924 20:19:54.562750 19727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:54.563603 19727 log.cc:824] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:54.566007 19727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26: No bootstrap required, opened a new log
Sep 24 20:19:54 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:54.570801 19727 raft_consensus.cc:348] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } }
I20260924 20:19:54.571080 19727 raft_consensus.cc:374] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:54.571168 19727 raft_consensus.cc:729] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 653d896e685f49308e5e5e1c5f23ae26, State: Initialized, Role: FOLLOWER
I20260924 20:19:54.571656 19727 consensus_queue.cc:260] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } }
I20260924 20:19:54.571825 19727 raft_consensus.cc:388] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:54.571904 19727 raft_consensus.cc:482] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:54.572027 19727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:54.578282 19730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:54.559312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.18.172.126:43793)
0924 20:19:54.559749 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:54.559766 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:54.559803 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:19:54.560796 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:54.560823 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:54.561287 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:54.561667 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:54.562540 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:54.564784 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:54.566023 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:54.566573 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:54.566694 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:54.569534 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:54.569543 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:54.569565 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:54.569599 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:54.575351 (+ 5752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:54.575837 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:54.575843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:54.575848 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:54.575967 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:54.576216 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:54.576218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:54.576219 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:54.576659 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:54.576816 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:54.577054 (+ 238us) server_negotiation.cc:299] Negotiation successful
0924 20:19:54.577326 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":147,"threads_started":1}
I20260924 20:19:54.663712 19727 raft_consensus.cc:504] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } }
I20260924 20:19:54.664260 19727 leader_election.cc:302] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 653d896e685f49308e5e5e1c5f23ae26; no voters:
I20260924 20:19:54.664772 19727 leader_election.cc:288] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:54.664912 19732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:54.665165 19732 raft_consensus.cc:686] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [term 1 LEADER]: Becoming Leader. State: Replica: 653d896e685f49308e5e5e1c5f23ae26, State: Running, Role: LEADER
I20260924 20:19:54.665669 19732 consensus_queue.cc:237] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } }
I20260924 20:19:54.666754 19727 sys_catalog.cc:564] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:54.667639 19733 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "653d896e685f49308e5e5e1c5f23ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } } }
I20260924 20:19:54.667798 19733 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:54.668233 19734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 653d896e685f49308e5e5e1c5f23ae26. Latest consensus state: current_term: 1 leader_uuid: "653d896e685f49308e5e5e1c5f23ae26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "653d896e685f49308e5e5e1c5f23ae26" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43793 } } }
I20260924 20:19:54.668391 19734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:54.669390 19740 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:54.672701 19740 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:54.678073 19740 catalog_manager.cc:1347] Generated new cluster ID: 94e5a557814743bc97166a9478b5f562
I20260924 20:19:54.678148 19740 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:54.696586 19740 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:54.697683 19740 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:54.705497 19740 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 653d896e685f49308e5e5e1c5f23ae26: Generated new TSK 0
I20260924 20:19:54.706377 19740 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
W20260924 20:19:55.418344 19121 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.172.65: real 0.708s user 0.000s sys 0.003s
I20260924 20:19:55.418883 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:55.536537 19753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:55.536826 19753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:55.536899 19753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:55.541152 19753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:55.541218 19753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:55.541306 19753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:19:55.547147 19753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:55.548470 19753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:55.549616 19753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:55.561710 19758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:55.561838 19761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:55.562033 19759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:55.562026 19753 server_base.cc:1034] running on GCE node
I20260924 20:19:55.562479 19753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:55.563266 19753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:55.564455 19753 hybrid_clock.cc:648] HybridClock initialized: now 1790281195564441 us; error 30 us; skew 500 ppm
Sep 24 20:19:55 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:55.567428 19753 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:19:55.568499 19753 webserver.cc:460] Webserver started at http://127.18.172.65:35929/ using document root <none> and password file <none>
I20260924 20:19:55.569100 19753 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:55.569164 19753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:55.569350 19753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:55.571480 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "23c6d583709d48608723245230d0691c"
format_stamp: "Formatted at 2026-09-24 20:19:55 on dist-test-slave-m1qq"
server_key: "25a98ce98f58bbbd8cbef1fc86e153ee"
server_key_iv: "5bb1227412f7df71e921e4a5c285280f"
server_key_version: "encryptionkey@0"
I20260924 20:19:55.572003 19753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "23c6d583709d48608723245230d0691c"
format_stamp: "Formatted at 2026-09-24 20:19:55 on dist-test-slave-m1qq"
server_key: "25a98ce98f58bbbd8cbef1fc86e153ee"
server_key_iv: "5bb1227412f7df71e921e4a5c285280f"
server_key_version: "encryptionkey@0"
I20260924 20:19:55.576313 19753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:19:55.578704 19768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:55.580039 19753 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:19:55.580138 19753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "23c6d583709d48608723245230d0691c"
format_stamp: "Formatted at 2026-09-24 20:19:55 on dist-test-slave-m1qq"
server_key: "25a98ce98f58bbbd8cbef1fc86e153ee"
server_key_iv: "5bb1227412f7df71e921e4a5c285280f"
server_key_version: "encryptionkey@0"
I20260924 20:19:55.580266 19753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:55.603428 19753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:55.606685 19753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:55.606856 19753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:55.607504 19753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:55.608400 19753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:55.608451 19753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:55.608495 19753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:55.608512 19753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:55.618961 19753 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:40813
I20260924 20:19:55.619045 19881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:40813 every 8 connection(s)
I20260924 20:19:55.620157 19753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:19:55.625764 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19753
I20260924 20:19:55.625887 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:19:55.626147 19121 external_mini_cluster.cc:1428] Setting key 0f83a6c3a5729197a694dbd6accb79c4
Sep 24 20:19:55 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281195, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:55.636613 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:55.622302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54259 (local address 127.18.172.126:43793)
0924 20:19:55.622580 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:55.622586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:55.623618 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 20:19:55.624870 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:55.624873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:55.624928 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:55.625050 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:55.626788 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:55.627404 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:55.629134 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:55.629381 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:55.629535 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:55.632374 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:55.632381 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:55.632386 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:55.632422 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:55.634102 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:55.634792 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:55.634795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:55.634796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:55.634876 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:55.635313 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:55.635316 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:55.635317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:55.635511 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:55.635589 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:55.636295 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 20:19:55.636445 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:19:55.637665 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:55.622825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43793 (local address 127.18.172.65:54259)
0924 20:19:55.623456 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:55.623502 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:55.624663 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:55.625226 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:55.625237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:55.625858 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:55.626561 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:55.626588 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:55.627531 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:55.627535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:55.628991 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:55.629006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:55.629562 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:55.629567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:55.629770 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:55.630417 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:55.630434 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:55.632160 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:55.634246 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:55.634253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:55.634271 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:55.634661 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:55.634973 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:55.634977 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:55.634979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:55.635207 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:55.635622 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:55.635630 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:55.636005 (+ 375us) client_negotiation.cc:769] Sending connection context
0924 20:19:55.636260 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 20:19:55.636652 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":154,"threads_started":1}
I20260924 20:19:55.638962 19882 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43793
I20260924 20:19:55.639319 19882 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:55.639865 19882 heartbeater.cc:511] Master 127.18.172.126:43793 requested a full tablet report, sending...
I20260924 20:19:55.641480 19691 ts_manager.cc:194] Registered new tserver with Master: 23c6d583709d48608723245230d0691c (127.18.172.65:40813)
I20260924 20:19:55.642660 19691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:54259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
W20260924 20:19:56.274950 19121 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.172.66: real 0.647s user 0.000s sys 0.003s
I20260924 20:19:56.275614 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:56.395407 19888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:56.395684 19888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:56.395730 19888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:56.399992 19888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:56.400064 19888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:56.400135 19888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:19:56.405737 19888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:56.406869 19888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:56.407996 19888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:56.418009 19896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:56.417990 19893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:56.418148 19894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:56.418286 19888 server_base.cc:1034] running on GCE node
I20260924 20:19:56.418792 19888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:56.419550 19888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:56.420912 19888 hybrid_clock.cc:648] HybridClock initialized: now 1790281196420893 us; error 29 us; skew 500 ppm
Sep 24 20:19:56 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281196, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:56.424353 19888 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:19:56.425400 19888 webserver.cc:460] Webserver started at http://127.18.172.66:36467/ using document root <none> and password file <none>
I20260924 20:19:56.426111 19888 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:56.426208 19888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:56.426421 19888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:56.428761 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "cb644417230841d8ad212719140622e9"
format_stamp: "Formatted at 2026-09-24 20:19:56 on dist-test-slave-m1qq"
server_key: "4ee524a1f8472ed861ad62c9551f4b4f"
server_key_iv: "cb8d374a7a4d11e66a1cc9d4d238ef04"
server_key_version: "encryptionkey@0"
I20260924 20:19:56.429329 19888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "cb644417230841d8ad212719140622e9"
format_stamp: "Formatted at 2026-09-24 20:19:56 on dist-test-slave-m1qq"
server_key: "4ee524a1f8472ed861ad62c9551f4b4f"
server_key_iv: "cb8d374a7a4d11e66a1cc9d4d238ef04"
server_key_version: "encryptionkey@0"
I20260924 20:19:56.433262 19888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:19:56.435985 19903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:56.437347 19888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:56.437480 19888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "cb644417230841d8ad212719140622e9"
format_stamp: "Formatted at 2026-09-24 20:19:56 on dist-test-slave-m1qq"
server_key: "4ee524a1f8472ed861ad62c9551f4b4f"
server_key_iv: "cb8d374a7a4d11e66a1cc9d4d238ef04"
server_key_version: "encryptionkey@0"
I20260924 20:19:56.437613 19888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:56.456549 19888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:56.460235 19888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:56.460448 19888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:56.461100 19888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:56.461978 19888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:56.462060 19888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:56.462122 19888 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:56.462188 19888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:56.472604 19888 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36951
I20260924 20:19:56.472700 20016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36951 every 8 connection(s)
I20260924 20:19:56.473584 19888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:19:56.483767 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 19888
I20260924 20:19:56.483914 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:19:56.484266 19121 external_mini_cluster.cc:1428] Setting key 64cf0e8bd26d04f24b8748e37f356165
Sep 24 20:19:56 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281196, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:56.489238 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:56.475671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:40605 (local address 127.18.172.126:43793)
0924 20:19:56.477177 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:56.477183 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:56.477204 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:19:56.478363 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:56.478370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:56.478470 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:56.478609 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:56.479907 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:56.480478 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:56.481918 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:56.482179 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:56.482298 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:56.485247 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:56.485251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:56.485254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:56.485278 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:56.487114 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:56.487660 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:56.487664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:56.487666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:56.487752 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:56.488100 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:56.488104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:56.488106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:56.488305 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:56.488410 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:56.488920 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 20:19:56.489048 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1401,"thread_start_us":58,"threads_started":1}
I20260924 20:19:56.490087 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:56.476086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43793 (local address 127.18.172.66:40605)
0924 20:19:56.476573 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:56.476628 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:56.478035 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:56.478797 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:56.478804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:56.479189 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:56.479747 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:56.479763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:56.480636 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:56.480639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:56.481772 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:56.481782 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:56.482350 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:56.482354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:56.482533 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:56.483316 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:56.483332 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:56.485085 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:56.487251 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:56.487257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:56.487274 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:56.487549 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:56.487845 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:56.487849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:56.487851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:56.488002 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:56.488409 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:56.488417 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:56.488679 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 20:19:56.488881 (+ 202us) client_negotiation.cc:240] Negotiation successful
0924 20:19:56.489235 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":126,"threads_started":1}
I20260924 20:19:56.491394 20017 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43793
I20260924 20:19:56.491693 20017 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:56.492350 20017 heartbeater.cc:511] Master 127.18.172.126:43793 requested a full tablet report, sending...
I20260924 20:19:56.493664 19691 ts_manager.cc:194] Registered new tserver with Master: cb644417230841d8ad212719140622e9 (127.18.172.66:36951)
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:19:56.494490 19691 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:40605
I20260924 20:19:56.645704 19882 heartbeater.cc:503] Master 127.18.172.126:43793 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
W20260924 20:19:57.083333 19121 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.18.172.67: real 0.597s user 0.000s sys 0.004s
I20260924 20:19:57.084003 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:57.195848 20023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:57.196105 20023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:57.196149 20023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:57.200093 20023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:57.200145 20023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:57.200202 20023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:19:57.205324 20023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43793
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:57.206305 20023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:57.207317 20023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:19:57.216835 20023 server_base.cc:1034] running on GCE node
W20260924 20:19:57.216952 20031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:57.216784 20029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:57.216801 20028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:57.217494 20023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:57.218169 20023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:57.219331 20023 hybrid_clock.cc:648] HybridClock initialized: now 1790281197219296 us; error 46 us; skew 500 ppm
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:57.221975 20023 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:19:57.222847 20023 webserver.cc:460] Webserver started at http://127.18.172.67:45875/ using document root <none> and password file <none>
I20260924 20:19:57.223456 20023 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:57.223546 20023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:57.223775 20023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:57.225968 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "086586652ad643eab974507461a54048"
format_stamp: "Formatted at 2026-09-24 20:19:57 on dist-test-slave-m1qq"
server_key: "1f68b93ff2399b7425496bc11f902a44"
server_key_iv: "4cc02ae86d2394465b5e1f6d00cefa82"
server_key_version: "encryptionkey@0"
I20260924 20:19:57.226511 20023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "086586652ad643eab974507461a54048"
format_stamp: "Formatted at 2026-09-24 20:19:57 on dist-test-slave-m1qq"
server_key: "1f68b93ff2399b7425496bc11f902a44"
server_key_iv: "4cc02ae86d2394465b5e1f6d00cefa82"
server_key_version: "encryptionkey@0"
I20260924 20:19:57.230324 20023 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:19:57.232867 20038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:57.234149 20023 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:19:57.234282 20023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "086586652ad643eab974507461a54048"
format_stamp: "Formatted at 2026-09-24 20:19:57 on dist-test-slave-m1qq"
server_key: "1f68b93ff2399b7425496bc11f902a44"
server_key_iv: "4cc02ae86d2394465b5e1f6d00cefa82"
server_key_version: "encryptionkey@0"
I20260924 20:19:57.234396 20023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:57.254745 20023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:57.257855 20023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:57.258054 20023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:57.258641 20023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:19:57.259619 20023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:57.259675 20023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:57.259743 20023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:57.259778 20023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:57.268610 20023 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:34155
I20260924 20:19:57.268630 20151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:34155 every 8 connection(s)
I20260924 20:19:57.269512 20023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:19:57.270522 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20023
I20260924 20:19:57.270648 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:19:57.270932 19121 external_mini_cluster.cc:1428] Setting key 35429315d813b15e0f6341eb35ba006e
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:57.283296 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.271557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:34791 (local address 127.18.172.126:43793)
0924 20:19:57.271825 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.271829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.272837 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 20:19:57.274182 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.274185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.274231 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.274322 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:57.275558 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.276104 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.277237 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.277447 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.277532 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.280041 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:57.280044 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:57.280046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:57.280069 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:57.281382 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.281853 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.281856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.281857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.281935 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.282231 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.282233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.282234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.282404 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:57.282478 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.282954 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.283124 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260924 20:19:57.284245 20155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.272099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43793 (local address 127.18.172.67:34791)
0924 20:19:57.272633 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.272722 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.273968 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.274486 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.274492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.274870 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:57.275425 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.275439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.276226 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.276228 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.277132 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.277141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.277578 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.277580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.277742 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.278282 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:57.278323 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:57.279890 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:57.281483 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:57.281488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:57.281501 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:57.281760 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:57.282015 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:57.282017 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:57.282018 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:57.282149 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:57.282489 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:57.282494 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:57.282736 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.282892 (+ 156us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.283208 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":136,"threads_started":1}
I20260924 20:19:57.285333 20152 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43793
I20260924 20:19:57.285533 20152 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:57.285952 20152 heartbeater.cc:511] Master 127.18.172.126:43793 requested a full tablet report, sending...
I20260924 20:19:57.286857 19690 ts_manager.cc:194] Registered new tserver with Master: 086586652ad643eab974507461a54048 (127.18.172.67:34155)
I20260924 20:19:57.287443 19690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:34791
I20260924 20:19:57.296684 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:19:57.307055 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.300193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.18.172.126:43793)
0924 20:19:57.300322 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.300326 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.300480 (+ 154us) server_negotiation.cc:407] Connection header received
0924 20:19:57.300564 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.300566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.300601 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.300668 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:57.301331 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.301840 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.302636 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.302862 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.302940 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.304265 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:57.304268 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:57.304270 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:57.304291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:57.305695 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.306035 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.306037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.306039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.306109 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.306317 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.306319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.306320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.306512 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:57.306612 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.306799 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.306890 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:19:57.311837 19690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:19:57.314762 19690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:19:57.329437 20165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.323429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34155 (local address 127.0.0.1:40122)
0924 20:19:57.323818 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.323875 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.324010 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.324793 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.324799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.324819 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:57.324960 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.324968 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.326431 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.326438 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.327977 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.327986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.328883 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.328887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.328995 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.329037 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.329173 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.329293 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":104,"threads_started":1}
I20260924 20:19:57.329444 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.323607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40122 (local address 127.18.172.67:34155)
0924 20:19:57.323974 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.323987 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.324011 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:19:57.324374 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.324381 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.324611 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.325124 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:57.325300 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.326309 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.328111 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.328777 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.328877 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.329017 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.329206 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.329298 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":155,"threads_started":1}
I20260924 20:19:57.330247 20169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.324488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36951 (local address 127.0.0.1:40192)
0924 20:19:57.325189 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.325203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.325284 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.325884 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.325888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.325907 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:57.325977 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.325982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.327454 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.327457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.328674 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.328685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.329829 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.329833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.329922 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.329946 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.330011 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.330091 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"mutex_wait_us":59,"thread_start_us":78,"threads_started":1}
I20260924 20:19:57.330317 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.324642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.18.172.66:36951)
0924 20:19:57.325461 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.325466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.325483 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:19:57.325538 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.325541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.325740 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.326141 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:57.326265 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.327345 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.328822 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.329690 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.329826 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.329963 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.330065 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.330181 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":80,"threads_started":1}
I20260924 20:19:57.330928 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.324349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.18.172.65:40813)
0924 20:19:57.324743 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.324963 (+ 220us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.325288 (+ 325us) server_negotiation.cc:407] Connection header received
0924 20:19:57.325365 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.325369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.325550 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.325714 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:57.325946 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.327216 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.328767 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.329607 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.329727 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.329870 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.330147 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.330244 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":90,"threads_started":1}
I20260924 20:19:57.330247 20167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.324181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:40813 (local address 127.0.0.1:41056)
0924 20:19:57.325098 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.325111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.325193 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.325688 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.325691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.325707 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:57.325815 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.325824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.327342 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.327346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.328642 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.328651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.329923 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.329926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.330015 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.330033 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.330099 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.330149 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":31,"thread_start_us":80,"threads_started":1}
I20260924 20:19:57.333380 20086 tablet_service.cc:1467] Processing CreateTablet for tablet f16ca76ced0b4e4aba7ca3fac050a44d (DEFAULT_TABLE table=test-table [id=67265559fb6b46a5ab4f9f147ec183ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:57.333380 19951 tablet_service.cc:1467] Processing CreateTablet for tablet f16ca76ced0b4e4aba7ca3fac050a44d (DEFAULT_TABLE table=test-table [id=67265559fb6b46a5ab4f9f147ec183ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:57.334564 19951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16ca76ced0b4e4aba7ca3fac050a44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:57.334574 20086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16ca76ced0b4e4aba7ca3fac050a44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:57.336758 19816 tablet_service.cc:1467] Processing CreateTablet for tablet f16ca76ced0b4e4aba7ca3fac050a44d (DEFAULT_TABLE table=test-table [id=67265559fb6b46a5ab4f9f147ec183ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:57.337879 19816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16ca76ced0b4e4aba7ca3fac050a44d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:57.342344 20171 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Bootstrap starting.
I20260924 20:19:57.343091 20172 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Bootstrap starting.
I20260924 20:19:57.344611 20173 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Bootstrap starting.
I20260924 20:19:57.345317 20171 tablet_bootstrap.cc:654] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:57.345692 20172 tablet_bootstrap.cc:654] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:57.346118 20171 log.cc:824] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:57.346472 20172 log.cc:824] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:57.346748 20173 tablet_bootstrap.cc:654] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:57.347571 20173 log.cc:824] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:57.348328 20171 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: No bootstrap required, opened a new log
I20260924 20:19:57.348328 20172 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: No bootstrap required, opened a new log
I20260924 20:19:57.348529 20172 ts_tablet_manager.cc:1397] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:19:57.348536 20171 ts_tablet_manager.cc:1397] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:19:57.349473 20173 tablet_bootstrap.cc:492] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: No bootstrap required, opened a new log
I20260924 20:19:57.349654 20173 ts_tablet_manager.cc:1397] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260924 20:19:57.352633 20172 raft_consensus.cc:348] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.352931 20172 raft_consensus.cc:374] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:57.352998 20172 raft_consensus.cc:729] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 086586652ad643eab974507461a54048, State: Initialized, Role: FOLLOWER
I20260924 20:19:57.352972 20173 raft_consensus.cc:348] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.352984 20171 raft_consensus.cc:348] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.353245 20171 raft_consensus.cc:374] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:57.353246 20173 raft_consensus.cc:374] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:57.353299 20171 raft_consensus.cc:729] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb644417230841d8ad212719140622e9, State: Initialized, Role: FOLLOWER
I20260924 20:19:57.353302 20173 raft_consensus.cc:729] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6d583709d48608723245230d0691c, State: Initialized, Role: FOLLOWER
I20260924 20:19:57.353519 20172 consensus_queue.cc:260] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.353754 20173 consensus_queue.cc:260] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.353789 20171 consensus_queue.cc:260] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.354272 20152 heartbeater.cc:503] Master 127.18.172.126:43793 was elected leader, sending a full tablet report...
I20260924 20:19:57.354586 20172 ts_tablet_manager.cc:1428] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:19:57.354588 20171 ts_tablet_manager.cc:1428] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 20:19:57.354714 20173 ts_tablet_manager.cc:1428] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260924 20:19:57.354913 20017 heartbeater.cc:503] Master 127.18.172.126:43793 was elected leader, sending a full tablet report...
W20260924 20:19:57.372489 19883 tablet.cc:2369] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:19:57.475232 20018 tablet.cc:2369] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:19:57.481263 20178 raft_consensus.cc:482] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:19:57.481474 20178 raft_consensus.cc:504] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.482628 20178 leader_election.cc:288] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 086586652ad643eab974507461a54048 (127.18.172.67:34155), cb644417230841d8ad212719140622e9 (127.18.172.66:36951)
I20260924 20:19:57.487713 20170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.483688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:34077 (local address 127.18.172.66:36951)
0924 20:19:57.483863 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.483869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.483882 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:19:57.483937 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.483943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.483984 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.484057 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:57.484360 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.484922 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.486700 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.487183 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.487287 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.487329 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.487518 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.487594 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:19:57.487777 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.483419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36951 (local address 127.18.172.65:34077)
0924 20:19:57.483719 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.483733 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.483861 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.484092 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.484094 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.484127 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:57.484250 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.484257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.485042 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.485045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.486538 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.486548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.487300 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.487302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.487383 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.487402 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.487488 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.487562 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":156,"thread_start_us":106,"threads_started":1}
I20260924 20:19:57.488355 20180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.482950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34155 (local address 127.18.172.65:33817)
0924 20:19:57.483298 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:57.483315 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:57.483433 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:57.483891 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:57.483896 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:57.483917 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:57.484055 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.484062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.484985 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.484992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.486540 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:57.486548 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.487952 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.487955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:57.488058 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.488076 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:19:57.488134 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 20:19:57.488188 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":116,"thread_start_us":166,"threads_started":1}
I20260924 20:19:57.488368 20166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.483139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33817 (local address 127.18.172.67:34155)
0924 20:19:57.483285 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.483288 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.483406 (+ 118us) server_negotiation.cc:407] Connection header received
0924 20:19:57.483601 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.483605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.483658 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.483713 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:57.484168 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.484858 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.487377 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.487819 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.487959 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.488000 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.488154 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.488269 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:19:57.488531 19971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16ca76ced0b4e4aba7ca3fac050a44d" candidate_uuid: "23c6d583709d48608723245230d0691c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb644417230841d8ad212719140622e9" is_pre_election: true
I20260924 20:19:57.488837 19971 raft_consensus.cc:2454] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c6d583709d48608723245230d0691c in term 0.
I20260924 20:19:57.489105 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16ca76ced0b4e4aba7ca3fac050a44d" candidate_uuid: "23c6d583709d48608723245230d0691c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086586652ad643eab974507461a54048" is_pre_election: true
I20260924 20:19:57.489351 19772 leader_election.cc:302] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c6d583709d48608723245230d0691c, cb644417230841d8ad212719140622e9; no voters:
I20260924 20:19:57.489408 20106 raft_consensus.cc:2454] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c6d583709d48608723245230d0691c in term 0.
I20260924 20:19:57.489656 20178 raft_consensus.cc:2781] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:19:57.489792 20178 raft_consensus.cc:482] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:19:57.489861 20178 raft_consensus.cc:3037] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:57.490891 20178 raft_consensus.cc:504] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.491264 20178 leader_election.cc:288] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [CANDIDATE]: Term 1 election: Requested vote from peers 086586652ad643eab974507461a54048 (127.18.172.67:34155), cb644417230841d8ad212719140622e9 (127.18.172.66:36951)
I20260924 20:19:57.491674 19971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16ca76ced0b4e4aba7ca3fac050a44d" candidate_uuid: "23c6d583709d48608723245230d0691c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb644417230841d8ad212719140622e9"
I20260924 20:19:57.491707 20106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16ca76ced0b4e4aba7ca3fac050a44d" candidate_uuid: "23c6d583709d48608723245230d0691c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "086586652ad643eab974507461a54048"
I20260924 20:19:57.491809 19971 raft_consensus.cc:3037] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:57.491825 20106 raft_consensus.cc:3037] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:57.492743 19971 raft_consensus.cc:2454] T f16ca76ced0b4e4aba7ca3fac050a44d P cb644417230841d8ad212719140622e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c6d583709d48608723245230d0691c in term 1.
I20260924 20:19:57.492911 20106 raft_consensus.cc:2454] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c6d583709d48608723245230d0691c in term 1.
I20260924 20:19:57.493065 19772 leader_election.cc:302] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c6d583709d48608723245230d0691c, cb644417230841d8ad212719140622e9; no voters:
I20260924 20:19:57.493278 20178 raft_consensus.cc:2781] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:57.493527 20178 raft_consensus.cc:686] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [term 1 LEADER]: Becoming Leader. State: Replica: 23c6d583709d48608723245230d0691c, State: Running, Role: LEADER
I20260924 20:19:57.493889 20178 consensus_queue.cc:237] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } }
I20260924 20:19:57.497043 19690 catalog_manager.cc:5495] T f16ca76ced0b4e4aba7ca3fac050a44d P 23c6d583709d48608723245230d0691c reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c6d583709d48608723245230d0691c (127.18.172.65). New cstate: current_term: 1 leader_uuid: "23c6d583709d48608723245230d0691c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "086586652ad643eab974507461a54048" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c6d583709d48608723245230d0691c" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb644417230841d8ad212719140622e9" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36951 } health_report { overall_health: UNKNOWN } } }
W20260924 20:19:57.520457 20153 tablet.cc:2369] T f16ca76ced0b4e4aba7ca3fac050a44d P 086586652ad643eab974507461a54048: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:19:57.525555 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.516918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.18.172.65:40813)
0924 20:19:57.517060 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.517063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.517105 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:19:57.517232 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.517235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.517275 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.517342 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:57.518031 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.518525 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.519404 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.519606 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.519685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.522094 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:57.522105 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:57.522120 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:57.522142 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:57.523880 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.524262 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.524268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.524269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.524347 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.524628 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.524630 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.524631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.524900 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:57.525051 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.525201 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.525296 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:19:57.553689 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.544983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.18.172.65:40813)
0924 20:19:57.545174 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.545178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.545263 (+ 85us) server_negotiation.cc:407] Connection header received
0924 20:19:57.545363 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.545365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.545409 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.545481 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:57.546442 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.546996 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.547868 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.548168 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.548252 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.550638 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:57.550641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:57.550643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:57.550673 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:57.552047 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.552461 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.552467 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.552469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.552545 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.552809 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.552813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.552815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.552982 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:57.553115 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.553397 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.553520 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[19646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:19:57.580993 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:57.572394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.18.172.65:40813)
0924 20:19:57.572526 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:57.572531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:57.572732 (+ 201us) server_negotiation.cc:407] Connection header received
0924 20:19:57.572854 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:57.572861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:57.572904 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:57.572993 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:57.573721 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.574228 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.575142 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:57.575428 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:57.575536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:57.578071 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:57.578074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:57.578076 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:57.578101 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:57.579450 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.579827 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.579829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.579830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.579898 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:57.580109 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:57.580111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:57.580112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:57.580269 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:57.580404 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:57.580706 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 20:19:57.580829 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 20:19:57.582335 19796 tablet_service.cc:2908] Rejecting scan request for tablet f16ca76ced0b4e4aba7ca3fac050a44d: Uninitialized: clean time has not yet been initialized
I20260924 20:19:57.584470 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19753
I20260924 20:19:57.590850 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19888
I20260924 20:19:57.596477 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20023
I20260924 20:19:57.601936 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 19659
2026-09-24T20:19:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (7060 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[20196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[20196](info): set up 2 sockets
Sep 24 20:19:57 dist-test-slave-m1qq krb5kdc[20196](info): commencing operation
krb5kdc: starting...
W20260924 20:19:58.655828 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:19:58 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:19:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:19:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:19:58.782843 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38925
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:58.897562 20209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:58.897835 20209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:58.897876 20209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:58.901963 20209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:19:58.902024 20209 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:19:58.902041 20209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:58.902060 20209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:58.902076 20209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:19:58.902091 20209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:19:58.907437 20209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:38925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:58.908738 20209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:58.909726 20209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:58.918357 20214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:58.918349 20215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:58.918537 20209 server_base.cc:1034] running on GCE node
W20260924 20:19:58.918349 20217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:58.919363 20209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:58.920502 20209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:58.921695 20209 hybrid_clock.cc:648] HybridClock initialized: now 1790281198921680 us; error 37 us; skew 500 ppm
Sep 24 20:19:58 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281198, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:58.924541 20209 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:19:58.925450 20209 webserver.cc:460] Webserver started at http://127.18.172.126:38849/ using document root <none> and password file <none>
I20260924 20:19:58.926028 20209 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:58.926115 20209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:58.926368 20209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:58.928457 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "64c41f3eed234566ba6e707641923d3f"
format_stamp: "Formatted at 2026-09-24 20:19:58 on dist-test-slave-m1qq"
server_key: "dfb880e55eabefb516a2f75140327976"
server_key_iv: "d01c014ae6eceef9b0bb54b91c847f58"
server_key_version: "encryptionkey@0"
I20260924 20:19:58.929028 20209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "64c41f3eed234566ba6e707641923d3f"
format_stamp: "Formatted at 2026-09-24 20:19:58 on dist-test-slave-m1qq"
server_key: "dfb880e55eabefb516a2f75140327976"
server_key_iv: "d01c014ae6eceef9b0bb54b91c847f58"
server_key_version: "encryptionkey@0"
I20260924 20:19:58.932842 20209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:19:58.935341 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:58.936753 20209 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:19:58.936895 20209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "64c41f3eed234566ba6e707641923d3f"
format_stamp: "Formatted at 2026-09-24 20:19:58 on dist-test-slave-m1qq"
server_key: "dfb880e55eabefb516a2f75140327976"
server_key_iv: "d01c014ae6eceef9b0bb54b91c847f58"
server_key_version: "encryptionkey@0"
I20260924 20:19:58.937008 20209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:58.950771 20209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:58.954175 20209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:58.954427 20209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:58.962054 20209 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:38925
I20260924 20:19:58.962119 20286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:38925 every 8 connection(s)
I20260924 20:19:58.963047 20209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:19:58.965921 20287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:58.969303 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20209
I20260924 20:19:58.969484 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:19:58.969870 19121 external_mini_cluster.cc:1428] Setting key f592aacf7481c59f3c88dd7b6a18535c
I20260924 20:19:58.971794 20287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Bootstrap starting.
I20260924 20:19:58.974454 20287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:58.975373 20287 log.cc:824] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:58.977263 20287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: No bootstrap required, opened a new log
Sep 24 20:19:58 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:58.981050 20287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } }
I20260924 20:19:58.981292 20287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:58.981372 20287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c41f3eed234566ba6e707641923d3f, State: Initialized, Role: FOLLOWER
I20260924 20:19:58.981810 20287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } }
I20260924 20:19:58.981958 20287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:19:58.982053 20287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:19:58.982173 20287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:58.983134 20287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } }
I20260924 20:19:58.983501 20287 leader_election.cc:302] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64c41f3eed234566ba6e707641923d3f; no voters:
I20260924 20:19:58.983987 20287 leader_election.cc:288] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:19:58.983989 20292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:58.984283 20292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [term 1 LEADER]: Becoming Leader. State: Replica: 64c41f3eed234566ba6e707641923d3f, State: Running, Role: LEADER
I20260924 20:19:58.985188 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:58.971464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.18.172.126:38925)
0924 20:19:58.971916 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:58.971935 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:58.971977 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:19:58.972803 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:58.972826 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:58.973267 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:58.973631 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:58.974278 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:58.975868 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:58.976737 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:58.977203 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:58.977368 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:58.979846 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:58.979850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:58.979863 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:58.979887 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:58.981844 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:58.982363 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:58.982368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:58.982374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:58.982487 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:58.982777 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:58.982782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:58.982783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:58.983234 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:58.983430 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:58.983733 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 20:19:58.984278 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":125,"threads_started":1}
I20260924 20:19:58.985250 20287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:19:58.984640 20292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } }
I20260924 20:19:58.987216 20294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64c41f3eed234566ba6e707641923d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } } }
I20260924 20:19:58.987375 20294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [sys.catalog]: This master's current role is: LEADER
I20260924 20:19:58.987480 20293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64c41f3eed234566ba6e707641923d3f. Latest consensus state: current_term: 1 leader_uuid: "64c41f3eed234566ba6e707641923d3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64c41f3eed234566ba6e707641923d3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38925 } } }
I20260924 20:19:58.988119 20293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f [sys.catalog]: This master's current role is: LEADER
W20260924 20:19:58.990065 20306 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:19:58.990154 20306 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:19:58.990228 20307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:19:58.990887 20307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:19:58.997030 20307 catalog_manager.cc:1347] Generated new cluster ID: 13bd02faf63c47c98ebcb81de2761401
I20260924 20:19:58.997099 20307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:19:59.015173 20307 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:19:59.016170 20307 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:19:59.026957 20307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Generated new TSK 0
I20260924 20:19:59.027737 20307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:19:59.067157 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38925
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:59.182240 20313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:59.182529 20313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:59.182629 20313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:59.186971 20313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:59.187045 20313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:59.187089 20313 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:19:59.187122 20313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:19:59.192435 20313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:59.193480 20313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:59.194439 20313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:19:59.204226 20313 server_base.cc:1034] running on GCE node
W20260924 20:19:59.204190 20319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.204360 20321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.204211 20318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:59.204847 20313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:59.205431 20313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:59.206596 20313 hybrid_clock.cc:648] HybridClock initialized: now 1790281199206567 us; error 44 us; skew 500 ppm
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:59.209384 20313 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:19:59.210230 20313 webserver.cc:460] Webserver started at http://127.18.172.65:38269/ using document root <none> and password file <none>
I20260924 20:19:59.210788 20313 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:59.210847 20313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:59.211122 20313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:59.213105 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "074b785f63034bebbb2cd00c99486fd1"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "c89fb6fe33df9b4d84ce1f8ce04d5045"
server_key_iv: "2e2002083316c4b2bb32cad4b1911a58"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.213645 20313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "074b785f63034bebbb2cd00c99486fd1"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "c89fb6fe33df9b4d84ce1f8ce04d5045"
server_key_iv: "2e2002083316c4b2bb32cad4b1911a58"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.217280 20313 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:19:59.219789 20328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.221038 20313 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:19:59.221199 20313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "074b785f63034bebbb2cd00c99486fd1"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "c89fb6fe33df9b4d84ce1f8ce04d5045"
server_key_iv: "2e2002083316c4b2bb32cad4b1911a58"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.221316 20313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:59.245545 20313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:59.249364 20313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:59.249699 20313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:59.251348 20313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:59.251426 20313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.251506 20313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:59.251554 20313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:59.264024 20313 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:45303
I20260924 20:19:59.264164 20443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:45303 every 8 connection(s)
I20260924 20:19:59.265288 20313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:19:59.267297 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.252374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46933 (local address 127.18.172.126:38925)
0924 20:19:59.252551 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.252556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.253475 (+ 919us) server_negotiation.cc:407] Connection header received
0924 20:19:59.254574 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.254578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.254641 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.254702 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.256304 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.256952 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.258383 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.258630 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.258767 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.262920 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.262924 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.262926 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.262957 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.264934 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.265505 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.265508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.265510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.265705 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.266114 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.266119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.266123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.266363 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.266469 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.266974 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.267159 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:19:59.268155 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.252808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.65:46933)
0924 20:19:59.253272 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.253324 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.254275 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.255196 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.255203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.255543 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.256149 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.256165 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.257080 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.257085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.258261 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.258270 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.258823 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.258827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.259055 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.259771 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.259798 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.262739 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.265086 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.265091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.265103 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.265403 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.265819 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.265822 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.265824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.266006 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.266488 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.266494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.266742 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.266915 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.267277 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":150,"threads_started":1}
I20260924 20:19:59.269676 20444 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38925
I20260924 20:19:59.269855 20444 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:59.270430 20444 heartbeater.cc:511] Master 127.18.172.126:38925 requested a full tablet report, sending...
I20260924 20:19:59.272138 20240 ts_manager.cc:194] Registered new tserver with Master: 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303)
I20260924 20:19:59.273550 20240 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:46933
I20260924 20:19:59.274408 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20313
I20260924 20:19:59.274534 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:19:59.274883 19121 external_mini_cluster.cc:1428] Setting key e2b59cd419f5b167aee435a6ca677a6f
I20260924 20:19:59.280056 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.271835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.65:36533)
0924 20:19:59.272123 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.272138 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.272252 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.272518 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.272521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.272752 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.272876 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.272884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.273711 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.273714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.274561 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.274581 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.275147 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.275150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.275273 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.275855 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.275876 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.276280 (+ 404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.278306 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.278312 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.278316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.278632 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.278909 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.278913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.278917 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.279044 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.279405 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.279408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.279520 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.279738 (+ 218us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.279896 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":11,"thread_start_us":122,"threads_started":1}
I20260924 20:19:59.280196 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.271913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:36533 (local address 127.18.172.126:38925)
0924 20:19:59.272045 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.272050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.272220 (+ 170us) server_negotiation.cc:407] Connection header received
0924 20:19:59.272351 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.272354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.272406 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.272468 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.272991 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.273578 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.274678 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.275023 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.275132 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.276385 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.276390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.276392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.276419 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.278180 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.278740 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.278742 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.278743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.278817 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.279124 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.279126 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.279127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.279304 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.279380 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.279842 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.279978 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:19:59.304397 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38925
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:59.414836 20453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:59.415117 20453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:59.415163 20453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:59.419127 20453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:59.419181 20453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:59.419220 20453 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:19:59.419253 20453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:19:59.424453 20453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:59.425611 20453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:59.426605 20453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:59.436695 20458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.436738 20461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.436764 20459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:59.436877 20453 server_base.cc:1034] running on GCE node
I20260924 20:19:59.437373 20453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:59.438019 20453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:59.439178 20453 hybrid_clock.cc:648] HybridClock initialized: now 1790281199439161 us; error 28 us; skew 500 ppm
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:59.442119 20453 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:19:59.443063 20453 webserver.cc:460] Webserver started at http://127.18.172.66:44725/ using document root <none> and password file <none>
I20260924 20:19:59.443657 20453 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:59.443728 20453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:59.443940 20453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:59.445976 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "85b69f0d18234e4fa7737000dcc05123"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "a83b298c91430a5ebf79155b97c7be0b"
server_key_iv: "e82ab29c87214167b6fc3a88e8a241c4"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.446549 20453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "85b69f0d18234e4fa7737000dcc05123"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "a83b298c91430a5ebf79155b97c7be0b"
server_key_iv: "e82ab29c87214167b6fc3a88e8a241c4"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.450402 20453 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:59.452920 20468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.454250 20453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:19:59.454370 20453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "85b69f0d18234e4fa7737000dcc05123"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "a83b298c91430a5ebf79155b97c7be0b"
server_key_iv: "e82ab29c87214167b6fc3a88e8a241c4"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.454479 20453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:59.472265 20453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:59.475621 20453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:59.475829 20453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:59.477313 20453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:59.477365 20453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.477432 20453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:59.477465 20453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:59.488195 20453 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36439
I20260924 20:19:59.489365 20453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:19:59.490231 20583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36439 every 8 connection(s)
I20260924 20:19:59.490779 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20453
I20260924 20:19:59.490899 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:19:59.491245 19121 external_mini_cluster.cc:1428] Setting key 821103a6bb69207495533f71bded9421
I20260924 20:19:59.492712 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.478317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:33897 (local address 127.18.172.126:38925)
0924 20:19:59.478475 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.478480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.479338 (+ 858us) server_negotiation.cc:407] Connection header received
0924 20:19:59.480497 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.480501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.480557 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.480637 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.482315 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.482976 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.484250 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.484548 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.484668 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.488132 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.488136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.488138 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.488168 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.490169 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.490841 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.490845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.490847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.490945 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.491439 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.491444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.491446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.491703 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.491787 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.492376 (+ 589us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.492545 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:19:59.493798 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.478684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.66:33897)
0924 20:19:59.479185 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.479234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.480251 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.480868 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.480880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.481382 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.482134 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.482157 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.483138 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.483141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.484078 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.484090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.484720 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.484725 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.484947 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.486227 (+ 1280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.486242 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.487946 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.490365 (+ 2419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.490370 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.490382 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.490720 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.491110 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.491113 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.491116 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.491297 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.491822 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.491855 (+ 33us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.492106 (+ 251us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.492305 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.492673 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":135,"threads_started":1}
I20260924 20:19:59.495621 20584 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38925
I20260924 20:19:59.495765 20584 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:59.496372 20584 heartbeater.cc:511] Master 127.18.172.126:38925 requested a full tablet report, sending...
I20260924 20:19:59.497560 20241 ts_manager.cc:194] Registered new tserver with Master: 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439)
I20260924 20:19:59.498514 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:33897
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:19:59.507406 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.497976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:45625 (local address 127.18.172.126:38925)
0924 20:19:59.498109 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.498113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.499075 (+ 962us) server_negotiation.cc:407] Connection header received
0924 20:19:59.499276 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.499280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.499327 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.499414 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.500027 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.500770 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.502270 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.502480 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.502681 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.504120 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.504123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.504125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.504147 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.505660 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.506049 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.506051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.506052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.506145 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.506416 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.506418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.506419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.506699 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.506778 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.507126 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.507238 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:19:59.507696 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.497929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.66:45625)
0924 20:19:59.498955 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.498974 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.499109 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.499461 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.499466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.499732 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.499891 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.499900 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.500917 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.500924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.502053 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.502066 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.502710 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.502715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.502846 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.503419 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.503441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.503974 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.505758 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.505761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.505763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.505962 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.506232 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.506234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.506236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.506313 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.506816 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.506818 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.506904 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.507080 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.507277 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:19:59.526119 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38925
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:19:59.643528 20593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:19:59.643837 20593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:19:59.643925 20593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:19:59.648775 20593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:19:59.648855 20593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:19:59.648901 20593 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:19:59.649021 20593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:19:59.654757 20593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:38925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:19:59.655875 20593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:19:59.656893 20593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:19:59.666512 20601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.666497 20598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:19:59.666642 20599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:19:59.666734 20593 server_base.cc:1034] running on GCE node
I20260924 20:19:59.667263 20593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:19:59.667857 20593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:19:59.669025 20593 hybrid_clock.cc:648] HybridClock initialized: now 1790281199669010 us; error 28 us; skew 500 ppm
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:19:59.677222 20593 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:19:59.678179 20593 webserver.cc:460] Webserver started at http://127.18.172.67:41039/ using document root <none> and password file <none>
I20260924 20:19:59.678769 20593 fs_manager.cc:359] Metadata directory not provided
I20260924 20:19:59.678830 20593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:19:59.679092 20593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:19:59.681192 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "0f4fb1bd26a24de49db2ae67c0697c39"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "87780a8d401192bb2c027bc8f1988242"
server_key_iv: "3b39bd685ac6c3e1142b8fd5d8311fb3"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.681758 20593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "0f4fb1bd26a24de49db2ae67c0697c39"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "87780a8d401192bb2c027bc8f1988242"
server_key_iv: "3b39bd685ac6c3e1142b8fd5d8311fb3"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.685783 20593 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:19:59.688534 20608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.689893 20593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:19:59.690024 20593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "0f4fb1bd26a24de49db2ae67c0697c39"
format_stamp: "Formatted at 2026-09-24 20:19:59 on dist-test-slave-m1qq"
server_key: "87780a8d401192bb2c027bc8f1988242"
server_key_iv: "3b39bd685ac6c3e1142b8fd5d8311fb3"
server_key_version: "encryptionkey@0"
I20260924 20:19:59.690138 20593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:19:59.702970 20593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:19:59.705945 20593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:19:59.706143 20593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:19:59.707707 20593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:19:59.707777 20593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:19:59.707849 20593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:19:59.707899 20593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:59.718137 20593 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:39553
I20260924 20:19:59.718209 20723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:39553 every 8 connection(s)
I20260924 20:19:59.719100 20593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:19:59.722429 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.708501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:53645 (local address 127.18.172.126:38925)
0924 20:19:59.708660 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.708665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.709526 (+ 861us) server_negotiation.cc:407] Connection header received
0924 20:19:59.710523 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.710535 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.710598 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.710682 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.712518 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.713416 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.715267 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.715706 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.715792 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.718667 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.718671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.718674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.718714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.720456 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.721106 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.721109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.721110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.721189 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.721484 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.721486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.721487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.721658 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.721733 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.722217 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.722311 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:19:59.722945 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20593
I20260924 20:19:59.723105 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:19:59.723372 19121 external_mini_cluster.cc:1428] Setting key ad5220a76a3bb891062851e2dbb2a868
I20260924 20:19:59.723687 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.708919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.67:53645)
0924 20:19:59.709363 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.709410 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.710246 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.711171 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.711183 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.711634 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.712338 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.712360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.713556 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.713559 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.715139 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.715159 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.715873 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.715877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.716112 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.716864 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.716891 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.718490 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.720727 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.720733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.720745 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.721010 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.721273 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.721275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.721277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.721405 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.721752 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.721757 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.721986 (+ 229us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.722146 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.722405 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":136,"threads_started":1}
I20260924 20:19:59.725332 20724 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38925
I20260924 20:19:59.725476 20724 heartbeater.cc:464] Registering TS with master...
I20260924 20:19:59.726073 20724 heartbeater.cc:511] Master 127.18.172.126:38925 requested a full tablet report, sending...
I20260924 20:19:59.727195 20241 ts_manager.cc:194] Registered new tserver with Master: 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553)
I20260924 20:19:59.727942 20241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:53645
I20260924 20:19:59.735103 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.727783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:50437 (local address 127.18.172.126:38925)
0924 20:19:59.727926 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.727931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.728093 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:19:59.728261 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.728264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.728308 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.728392 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.728890 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.729378 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.730441 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.730645 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.730730 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.732018 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.732021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.732023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.732043 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.733531 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.733951 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.733954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.733955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.734026 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.734302 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.734304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.734306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.734460 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.734534 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.734837 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.734925 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:19:59.735134 20730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.727672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.18.172.67:50437)
0924 20:19:59.727960 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.727982 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.728142 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.728419 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.728421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.728655 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:19:59.728779 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.728786 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.729567 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.729570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.730337 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.730345 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.730750 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.730753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.730823 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.731338 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:19:59.731351 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:19:59.731870 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:19:59.733645 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.733648 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.733650 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.733863 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.734133 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:19:59.734136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:19:59.734137 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:19:59.734218 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:19:59.734599 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:19:59.734601 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:19:59.734680 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.734777 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.734886 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":108,"threads_started":1}
I20260924 20:19:59.737344 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:19:59 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:19:59.762559 20290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.754724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33808 (local address 127.18.172.126:38925)
0924 20:19:59.754868 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.754871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.754885 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:19:59.754927 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.754929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.754969 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.755110 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:19:59.755851 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.756365 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.757236 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.757485 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.757580 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.759939 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:19:59.759941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:19:59.759943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:19:59.759965 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:19:59.761195 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.761566 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.761568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.761569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.761636 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:19:59.761869 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:19:59.761871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:19:59.761872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:19:59.762025 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:19:59.762119 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.762355 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.762440 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:19:59.766007 20241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:19:59.769493 20241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:19:59.785353 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.778600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53004 (local address 127.18.172.66:36439)
0924 20:19:59.779085 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.779100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.779148 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:19:59.779243 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.779251 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.779490 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.779704 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:59.780304 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.781785 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.783897 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.784621 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.784757 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.784935 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.785085 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.785186 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":111,"threads_started":1}
I20260924 20:19:59.785347 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.778394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36439 (local address 127.0.0.1:53004)
0924 20:19:59.778797 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.778855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.778991 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.779953 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.779960 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.779987 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:59.780129 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.780136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.781927 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.781931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.783793 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.783802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.784755 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.784759 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.784872 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.784912 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.785032 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.785212 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":307,"thread_start_us":110,"threads_started":1}
I20260924 20:19:59.785739 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.779278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53666 (local address 127.18.172.65:45303)
0924 20:19:59.780159 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.780168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.780194 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:19:59.780275 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.780281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.780482 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.780658 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:59.780942 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.781776 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.783851 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.784673 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.785207 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.785352 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.785407 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.785589 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":70,"threads_started":1}
I20260924 20:19:59.785808 20745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.779313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:39553 (local address 127.0.0.1:35216)
0924 20:19:59.779813 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.779828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.779910 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.780838 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.780841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.780859 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:59.780967 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.780976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.782361 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.782368 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.783659 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.783669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.785031 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.785035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.785126 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.785164 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.785233 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.785305 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260924 20:19:59.786077 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.779737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.18.172.67:39553)
0924 20:19:59.780360 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.780371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.780390 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:19:59.780457 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.780462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.780638 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.780782 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:59.781107 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.782248 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.783788 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.784673 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.784801 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.784968 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.785427 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.785942 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":121,"threads_started":1}
I20260924 20:19:59.786361 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.778820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45303 (local address 127.0.0.1:53666)
0924 20:19:59.779286 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.779300 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.779388 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.780650 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.780654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.780675 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:59.780803 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.780810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.781901 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.781908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.783702 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.783713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.784827 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.784831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.784926 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.784956 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.785013 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.786238 (+ 1225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"mutex_wait_us":1089,"thread_start_us":101,"threads_started":1}
I20260924 20:19:59.789027 20378 tablet_service.cc:1467] Processing CreateTablet for tablet cf732a25c8684ecf8144804b7fceea44 (DEFAULT_TABLE table=test-table [id=95381c3031014d249d3c6c8f96ff1a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.789073 20658 tablet_service.cc:1467] Processing CreateTablet for tablet cf732a25c8684ecf8144804b7fceea44 (DEFAULT_TABLE table=test-table [id=95381c3031014d249d3c6c8f96ff1a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.789077 20518 tablet_service.cc:1467] Processing CreateTablet for tablet cf732a25c8684ecf8144804b7fceea44 (DEFAULT_TABLE table=test-table [id=95381c3031014d249d3c6c8f96ff1a5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.789952 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf732a25c8684ecf8144804b7fceea44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.790109 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf732a25c8684ecf8144804b7fceea44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.790102 20518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf732a25c8684ecf8144804b7fceea44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.796872 20747 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Bootstrap starting.
I20260924 20:19:59.798686 20749 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Bootstrap starting.
I20260924 20:19:59.799345 20748 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Bootstrap starting.
I20260924 20:19:59.799438 20747 tablet_bootstrap.cc:654] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.800184 20747 log.cc:824] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:59.800921 20749 tablet_bootstrap.cc:654] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.801528 20748 tablet_bootstrap.cc:654] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.801844 20749 log.cc:824] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:59.802363 20748 log.cc:824] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Log is configured to *not* fsync() on all Append() calls
I20260924 20:19:59.803081 20747 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: No bootstrap required, opened a new log
I20260924 20:19:59.803279 20747 ts_tablet_manager.cc:1397] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:19:59.804087 20749 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: No bootstrap required, opened a new log
I20260924 20:19:59.804113 20748 tablet_bootstrap.cc:492] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: No bootstrap required, opened a new log
I20260924 20:19:59.804281 20749 ts_tablet_manager.cc:1397] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:19:59.804319 20748 ts_tablet_manager.cc:1397] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:19:59.807346 20748 raft_consensus.cc:348] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.807771 20748 raft_consensus.cc:374] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.807833 20748 raft_consensus.cc:729] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b69f0d18234e4fa7737000dcc05123, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.808012 20747 raft_consensus.cc:348] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.808296 20747 raft_consensus.cc:374] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.808365 20747 raft_consensus.cc:729] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074b785f63034bebbb2cd00c99486fd1, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.808403 20748 consensus_queue.cc:260] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.808785 20749 raft_consensus.cc:348] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.809051 20749 raft_consensus.cc:374] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.809111 20749 raft_consensus.cc:729] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4fb1bd26a24de49db2ae67c0697c39, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.809404 20747 consensus_queue.cc:260] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.809607 20748 ts_tablet_manager.cc:1428] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:19:59.809525 20749 consensus_queue.cc:260] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.810014 20584 heartbeater.cc:503] Master 127.18.172.126:38925 was elected leader, sending a full tablet report...
I20260924 20:19:59.810468 20444 heartbeater.cc:503] Master 127.18.172.126:38925 was elected leader, sending a full tablet report...
I20260924 20:19:59.810549 20749 ts_tablet_manager.cc:1428] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:19:59.810858 20724 heartbeater.cc:503] Master 127.18.172.126:38925 was elected leader, sending a full tablet report...
I20260924 20:19:59.810461 20747 ts_tablet_manager.cc:1428] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:19:59.901134 20755 raft_consensus.cc:482] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:19:59.901321 20755 raft_consensus.cc:504] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.902417 20755 leader_election.cc:288] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439), 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303)
I20260924 20:19:59.906709 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.902594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36439 (local address 127.18.172.67:45925)
0924 20:19:59.902749 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.902768 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.902861 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.903253 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.903258 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.903282 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:59.903410 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.903417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.904300 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.904304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.905192 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.905201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.906222 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.906225 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.906347 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.906399 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.906480 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.906563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":22}
I20260924 20:19:59.906734 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.902816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:45925 (local address 127.18.172.66:36439)
0924 20:19:59.902951 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.902957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.902972 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:19:59.903060 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.903063 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.903121 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.903209 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:59.903530 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.904190 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.905337 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.906020 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.906187 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.906248 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.906515 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.906597 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:19:59.907473 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf732a25c8684ecf8144804b7fceea44" candidate_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b69f0d18234e4fa7737000dcc05123" is_pre_election: true
I20260924 20:19:59.907676 20744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.902832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:32913 (local address 127.18.172.65:45303)
0924 20:19:59.902960 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.902965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.903242 (+ 277us) server_negotiation.cc:407] Connection header received
0924 20:19:59.903351 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.903354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.903404 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.903493 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:19:59.903889 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.904448 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.906439 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.907142 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.907260 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.907304 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.907480 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.907553 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:19:59.907750 20756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.902746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45303 (local address 127.18.172.67:32913)
0924 20:19:59.903144 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 20:19:59.903159 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:19:59.903255 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:19:59.903526 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:19:59.903529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:19:59.903545 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:19:59.903770 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.903778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.904586 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.904589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.906310 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:19:59.906319 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.907275 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.907279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:19:59.907357 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.907376 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 20:19:59.907443 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:19:59.907495 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":1,"thread_start_us":118,"threads_started":1}
I20260924 20:19:59.907814 20538 raft_consensus.cc:2454] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f4fb1bd26a24de49db2ae67c0697c39 in term 0.
I20260924 20:19:59.908334 20612 leader_election.cc:302] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f4fb1bd26a24de49db2ae67c0697c39, 85b69f0d18234e4fa7737000dcc05123; no voters:
I20260924 20:19:59.908380 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf732a25c8684ecf8144804b7fceea44" candidate_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074b785f63034bebbb2cd00c99486fd1" is_pre_election: true
I20260924 20:19:59.908601 20755 raft_consensus.cc:2781] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:19:59.908727 20755 raft_consensus.cc:482] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:19:59.908783 20755 raft_consensus.cc:3037] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:59.908882 20398 raft_consensus.cc:2454] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f4fb1bd26a24de49db2ae67c0697c39 in term 0.
I20260924 20:19:59.909973 20755 raft_consensus.cc:504] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.910365 20755 leader_election.cc:288] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [CANDIDATE]: Term 1 election: Requested vote from peers 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439), 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303)
I20260924 20:19:59.910802 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf732a25c8684ecf8144804b7fceea44" candidate_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074b785f63034bebbb2cd00c99486fd1"
I20260924 20:19:59.910826 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf732a25c8684ecf8144804b7fceea44" candidate_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b69f0d18234e4fa7737000dcc05123"
I20260924 20:19:59.910921 20398 raft_consensus.cc:3037] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:59.910953 20538 raft_consensus.cc:3037] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:19:59.911978 20398 raft_consensus.cc:2454] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f4fb1bd26a24de49db2ae67c0697c39 in term 1.
I20260924 20:19:59.912410 20609 leader_election.cc:302] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b785f63034bebbb2cd00c99486fd1, 0f4fb1bd26a24de49db2ae67c0697c39; no voters:
I20260924 20:19:59.912647 20755 raft_consensus.cc:2781] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:19:59.912688 20538 raft_consensus.cc:2454] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f4fb1bd26a24de49db2ae67c0697c39 in term 1.
I20260924 20:19:59.912916 20755 raft_consensus.cc:686] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 LEADER]: Becoming Leader. State: Replica: 0f4fb1bd26a24de49db2ae67c0697c39, State: Running, Role: LEADER
I20260924 20:19:59.913293 20755 consensus_queue.cc:237] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } }
I20260924 20:19:59.916939 20239 catalog_manager.cc:5495] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } health_report { overall_health: UNKNOWN } } }
I20260924 20:19:59.943974 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:19:59.940057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.18.172.67:39553)
0924 20:19:59.940192 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:19:59.940195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:19:59.940208 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:19:59.940386 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:19:59.940389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:19:59.940443 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:19:59.940527 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:19:59.940803 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.941349 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.942300 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:19:59.942506 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:19:59.942590 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:19:59.942909 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:19:59.943435 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:19:59.943518 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:19:59.943811 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 20:19:59.943868 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:19:59.950014 20398 raft_consensus.cc:1261] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Refusing update from remote peer 0f4fb1bd26a24de49db2ae67c0697c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:59.950062 20538 raft_consensus.cc:1261] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Refusing update from remote peer 0f4fb1bd26a24de49db2ae67c0697c39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:19:59.950745 20755 consensus_queue.cc:1035] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:59.950956 20757 consensus_queue.cc:1035] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:19:59.960031 20761 mvcc.cc:205] Tried to move back new op lower bound from 7332991794988306432 to 7332991794848256000. Current Snapshot: MvccSnapshot[applied={T|T < 7332991794988306432}]
I20260924 20:19:59.961849 20762 mvcc.cc:205] Tried to move back new op lower bound from 7332991794988306432 to 7332991794848256000. Current Snapshot: MvccSnapshot[applied={T|T < 7332991794988306432}]
I20260924 20:19:59.970623 20240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:19:59.970808 20240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:19:59.973095 20240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Sending DeleteTablet for 3 replicas of tablet cf732a25c8684ecf8144804b7fceea44
I20260924 20:19:59.973923 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet cf732a25c8684ecf8144804b7fceea44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:19:59 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:53666
I20260924 20:19:59.973927 20518 tablet_service.cc:1514] Processing DeleteTablet for tablet cf732a25c8684ecf8144804b7fceea44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:19:59 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:53004
I20260924 20:19:59.974555 20658 tablet_service.cc:1514] Processing DeleteTablet for tablet cf732a25c8684ecf8144804b7fceea44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:19:59 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35216
I20260924 20:19:59.974841 20239 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:19:59.975263 20239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:19:59.976075 20769 tablet_replica.cc:330] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: stopping tablet replica
I20260924 20:19:59.976445 20769 raft_consensus.cc:2229] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:19:59.976758 20769 raft_consensus.cc:2258] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:19:59.977114 20770 tablet_replica.cc:330] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: stopping tablet replica
I20260924 20:19:59.977418 20770 raft_consensus.cc:2229] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:19:59.977751 20770 raft_consensus.cc:2258] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:19:59.978094 20771 tablet_replica.cc:330] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: stopping tablet replica
I20260924 20:19:59.978369 20771 raft_consensus.cc:2229] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:19:59.978708 20771 raft_consensus.cc:2258] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:19:59.980914 20769 ts_tablet_manager.cc:1905] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:19:59.981806 20518 tablet_service.cc:1467] Processing CreateTablet for tablet 0c721d3dc4fd4846959a54170015a167 (DEFAULT_TABLE table=test-table [id=27a750078c774549bf7975209b6a97c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.982108 20518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c721d3dc4fd4846959a54170015a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.982543 20658 tablet_service.cc:1467] Processing CreateTablet for tablet 0c721d3dc4fd4846959a54170015a167 (DEFAULT_TABLE table=test-table [id=27a750078c774549bf7975209b6a97c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.982807 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c721d3dc4fd4846959a54170015a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.983206 20771 ts_tablet_manager.cc:1905] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:19:59.983234 20378 tablet_service.cc:1467] Processing CreateTablet for tablet 0c721d3dc4fd4846959a54170015a167 (DEFAULT_TABLE table=test-table [id=27a750078c774549bf7975209b6a97c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:19:59.983479 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c721d3dc4fd4846959a54170015a167. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:19:59.984720 20748 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Bootstrap starting.
I20260924 20:19:59.985517 20769 ts_tablet_manager.cc:1918] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:19:59.985724 20769 log.cc:1196] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/cf732a25c8684ecf8144804b7fceea44
I20260924 20:19:59.986047 20748 tablet_bootstrap.cc:654] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.988339 20771 ts_tablet_manager.cc:1918] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:19:59.988389 20770 ts_tablet_manager.cc:1905] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:19:59.988549 20771 log.cc:1196] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/cf732a25c8684ecf8144804b7fceea44
I20260924 20:19:59.988922 20771 ts_tablet_manager.cc:1939] T cf732a25c8684ecf8144804b7fceea44 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting consensus metadata
I20260924 20:19:59.989163 20769 ts_tablet_manager.cc:1939] T cf732a25c8684ecf8144804b7fceea44 P 074b785f63034bebbb2cd00c99486fd1: Deleting consensus metadata
I20260924 20:19:59.989521 20747 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Bootstrap starting.
I20260924 20:19:59.989861 20749 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Bootstrap starting.
I20260924 20:19:59.990527 20747 tablet_bootstrap.cc:654] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.990847 20749 tablet_bootstrap.cc:654] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Neither blocks nor log segments found. Creating new log.
I20260924 20:19:59.992157 20770 ts_tablet_manager.cc:1918] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:19:59.992257 20770 log.cc:1196] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/cf732a25c8684ecf8144804b7fceea44
I20260924 20:19:59.992599 20770 ts_tablet_manager.cc:1939] T cf732a25c8684ecf8144804b7fceea44 P 85b69f0d18234e4fa7737000dcc05123: Deleting consensus metadata
I20260924 20:19:59.993425 20748 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: No bootstrap required, opened a new log
I20260924 20:19:59.993512 20748 ts_tablet_manager.cc:1397] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260924 20:19:59.993706 20225 catalog_manager.cc:4841] TS 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303): tablet cf732a25c8684ecf8144804b7fceea44 (table test-table [id=95381c3031014d249d3c6c8f96ff1a5d]) successfully deleted
I20260924 20:19:59.994017 20748 raft_consensus.cc:348] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.994110 20228 catalog_manager.cc:4841] TS 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439): tablet cf732a25c8684ecf8144804b7fceea44 (table test-table [id=95381c3031014d249d3c6c8f96ff1a5d]) successfully deleted
I20260924 20:19:59.994141 20748 raft_consensus.cc:374] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.994195 20748 raft_consensus.cc:729] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b69f0d18234e4fa7737000dcc05123, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.994319 20748 consensus_queue.cc:260] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.994575 20228 catalog_manager.cc:4841] TS 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553): tablet cf732a25c8684ecf8144804b7fceea44 (table test-table [id=95381c3031014d249d3c6c8f96ff1a5d]) successfully deleted
I20260924 20:19:59.994608 20748 ts_tablet_manager.cc:1428] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:19:59.994798 20747 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: No bootstrap required, opened a new log
I20260924 20:19:59.994894 20747 ts_tablet_manager.cc:1397] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 20:19:59.995390 20747 raft_consensus.cc:348] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.995508 20747 raft_consensus.cc:374] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.995556 20747 raft_consensus.cc:729] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074b785f63034bebbb2cd00c99486fd1, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.995672 20747 consensus_queue.cc:260] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.995926 20747 ts_tablet_manager.cc:1428] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:19:59.997128 20749 tablet_bootstrap.cc:492] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: No bootstrap required, opened a new log
I20260924 20:19:59.997225 20749 ts_tablet_manager.cc:1397] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260924 20:19:59.997722 20749 raft_consensus.cc:348] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.997844 20749 raft_consensus.cc:374] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:19:59.997879 20749 raft_consensus.cc:729] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4fb1bd26a24de49db2ae67c0697c39, State: Initialized, Role: FOLLOWER
I20260924 20:19:59.998005 20749 consensus_queue.cc:260] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:19:59.998265 20749 ts_tablet_manager.cc:1428] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20260924 20:20:00.017194 20445 tablet.cc:2369] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:20:00.220772 20725 tablet.cc:2369] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:20:00.240860 20585 tablet.cc:2369] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:00.314298 20754 raft_consensus.cc:482] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:00.314464 20754 raft_consensus.cc:504] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.315543 20754 leader_election.cc:288] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439), 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553)
I20260924 20:20:00.319893 20776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.316297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:39553 (local address 127.18.172.65:43099)
0924 20:20:00.316726 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.316737 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.316838 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.317065 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.317074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.317092 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:00.317210 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.317217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.317969 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.317972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.318761 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.318774 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.319470 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.319474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.319576 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.319595 (+ 19us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.319655 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.319706 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":93,"threads_started":1}
I20260924 20:20:00.319913 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.316443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:43099 (local address 127.18.172.67:39553)
0924 20:20:00.316595 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.316599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.316829 (+ 230us) server_negotiation.cc:407] Connection header received
0924 20:20:00.316916 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.316920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.316977 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.317031 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:00.317315 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.317843 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.318883 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.319335 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.319469 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.319521 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.319705 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.319790 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:20:00.320009 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.315993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33611 (local address 127.18.172.66:36439)
0924 20:20:00.316147 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.316151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.316242 (+ 91us) server_negotiation.cc:407] Connection header received
0924 20:20:00.316426 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.316429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.316484 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.316553 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:00.316971 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.317653 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.318807 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.319312 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.319429 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.319483 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.319819 (+ 336us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.319890 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:20:00.320269 20775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.315875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36439 (local address 127.18.172.65:33611)
0924 20:20:00.316138 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.316153 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.316270 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.316709 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.316711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.316727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:00.316833 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.316840 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.317768 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.317771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.318671 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.318680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.319435 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.319439 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.319537 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.319692 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.319777 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.319840 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260924 20:20:00.320713 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c721d3dc4fd4846959a54170015a167" candidate_uuid: "074b785f63034bebbb2cd00c99486fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" is_pre_election: true
I20260924 20:20:00.321009 20678 raft_consensus.cc:2454] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074b785f63034bebbb2cd00c99486fd1 in term 0.
I20260924 20:20:00.321034 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c721d3dc4fd4846959a54170015a167" candidate_uuid: "074b785f63034bebbb2cd00c99486fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b69f0d18234e4fa7737000dcc05123" is_pre_election: true
I20260924 20:20:00.321190 20538 raft_consensus.cc:2454] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074b785f63034bebbb2cd00c99486fd1 in term 0.
I20260924 20:20:00.321507 20332 leader_election.cc:302] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b785f63034bebbb2cd00c99486fd1, 0f4fb1bd26a24de49db2ae67c0697c39; no voters:
I20260924 20:20:00.321811 20754 raft_consensus.cc:2781] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:00.321885 20754 raft_consensus.cc:482] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:00.321939 20754 raft_consensus.cc:3037] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.322916 20754 raft_consensus.cc:504] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.323272 20754 leader_election.cc:288] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [CANDIDATE]: Term 1 election: Requested vote from peers 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439), 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553)
I20260924 20:20:00.323750 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c721d3dc4fd4846959a54170015a167" candidate_uuid: "074b785f63034bebbb2cd00c99486fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4fb1bd26a24de49db2ae67c0697c39"
I20260924 20:20:00.323771 20538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c721d3dc4fd4846959a54170015a167" candidate_uuid: "074b785f63034bebbb2cd00c99486fd1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b69f0d18234e4fa7737000dcc05123"
I20260924 20:20:00.323844 20678 raft_consensus.cc:3037] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.323925 20538 raft_consensus.cc:3037] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.324882 20678 raft_consensus.cc:2454] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074b785f63034bebbb2cd00c99486fd1 in term 1.
I20260924 20:20:00.325151 20538 raft_consensus.cc:2454] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074b785f63034bebbb2cd00c99486fd1 in term 1.
I20260924 20:20:00.325172 20332 leader_election.cc:302] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b785f63034bebbb2cd00c99486fd1, 0f4fb1bd26a24de49db2ae67c0697c39; no voters:
I20260924 20:20:00.325358 20754 raft_consensus.cc:2781] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:00.325553 20754 raft_consensus.cc:686] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 1 LEADER]: Becoming Leader. State: Replica: 074b785f63034bebbb2cd00c99486fd1, State: Running, Role: LEADER
I20260924 20:20:00.325837 20754 consensus_queue.cc:237] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.328512 20239 catalog_manager.cc:5495] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "074b785f63034bebbb2cd00c99486fd1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:20:00 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281198, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:00.394372 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.385058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.18.172.126:38925)
0924 20:20:00.385367 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.385375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.385451 (+ 76us) server_negotiation.cc:407] Connection header received
0924 20:20:00.385711 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.385714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.385760 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.385852 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:00.386600 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.387283 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.388102 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.388313 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.388418 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.390742 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:00.390751 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:00.390753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:00.390772 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:00.392470 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.393011 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.393018 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.393020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.393104 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.393402 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.393405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.393407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.393593 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:00.393711 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.394036 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.394195 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260924 20:20:00.394372 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.385020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38925 (local address 127.0.0.1:33818)
0924 20:20:00.385372 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.385390 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.385527 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.385869 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.385872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.386374 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:00.386482 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.386489 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.387385 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.387390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.387976 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.387985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.388420 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.388423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.388512 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.389120 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:00.389132 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:00.390515 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:00.392621 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:00.392632 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:00.392639 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:00.392901 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:00.393204 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:00.393209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:00.393212 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:00.393300 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:00.393710 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:00.393719 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:00.393839 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.393960 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.394131 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":111,"threads_started":1}
I20260924 20:20:00.398833 20239 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:33818:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:20:00.399601 20239 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:00.404623 20518 tablet_service.cc:1467] Processing CreateTablet for tablet 81a98250c5d645bf90938f50546868fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65e26908a554488b5b56cc13a16e1e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:00.404652 20658 tablet_service.cc:1467] Processing CreateTablet for tablet 81a98250c5d645bf90938f50546868fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65e26908a554488b5b56cc13a16e1e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:00.404662 20378 tablet_service.cc:1467] Processing CreateTablet for tablet 81a98250c5d645bf90938f50546868fa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d65e26908a554488b5b56cc13a16e1e5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:00.405045 20518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a98250c5d645bf90938f50546868fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:00.405041 20378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a98250c5d645bf90938f50546868fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:00.405052 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a98250c5d645bf90938f50546868fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:00.407792 20748 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: Bootstrap starting.
I20260924 20:20:00.408322 20749 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: Bootstrap starting.
I20260924 20:20:00.408556 20747 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: Bootstrap starting.
I20260924 20:20:00.408897 20748 tablet_bootstrap.cc:654] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:00.409319 20749 tablet_bootstrap.cc:654] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:00.409591 20747 tablet_bootstrap.cc:654] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:00.410277 20748 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: No bootstrap required, opened a new log
I20260924 20:20:00.410382 20748 ts_tablet_manager.cc:1397] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:00.410504 20749 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: No bootstrap required, opened a new log
I20260924 20:20:00.410619 20749 ts_tablet_manager.cc:1397] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 20:20:00.410878 20747 tablet_bootstrap.cc:492] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: No bootstrap required, opened a new log
I20260924 20:20:00.410962 20747 ts_tablet_manager.cc:1397] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 20:20:00.410930 20748 raft_consensus.cc:348] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.411098 20748 raft_consensus.cc:374] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:00.411156 20748 raft_consensus.cc:729] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b69f0d18234e4fa7737000dcc05123, State: Initialized, Role: FOLLOWER
I20260924 20:20:00.411330 20748 consensus_queue.cc:260] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.411331 20749 raft_consensus.cc:348] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.411448 20749 raft_consensus.cc:374] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:00.411500 20749 raft_consensus.cc:729] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f4fb1bd26a24de49db2ae67c0697c39, State: Initialized, Role: FOLLOWER
I20260924 20:20:00.411464 20747 raft_consensus.cc:348] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.411574 20747 raft_consensus.cc:374] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:00.411623 20747 raft_consensus.cc:729] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074b785f63034bebbb2cd00c99486fd1, State: Initialized, Role: FOLLOWER
I20260924 20:20:00.411609 20753 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.411654 20749 consensus_queue.cc:260] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.411767 20753 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.411871 20755 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.412014 20755 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.412451 20748 ts_tablet_manager.cc:1428] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:00.411749 20747 consensus_queue.cc:260] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.412875 20747 ts_tablet_manager.cc:1428] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:00.411942 20749 ts_tablet_manager.cc:1428] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:00.412811 20754 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.413447 20754 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.497822 20753 raft_consensus.cc:482] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:00.497947 20753 raft_consensus.cc:504] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.499261 20753 leader_election.cc:288] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303), 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553)
I20260924 20:20:00.503599 20789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.499459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45303 (local address 127.18.172.66:54159)
0924 20:20:00.499746 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.499761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.499861 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.500369 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.500372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.500408 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:00.500536 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.500545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.501573 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.501577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.502391 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.502400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.503196 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.503200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.503299 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.503325 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.503396 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.503465 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":21,"thread_start_us":64,"threads_started":1}
I20260924 20:20:00.503870 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.499739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:54159 (local address 127.18.172.65:45303)
0924 20:20:00.500118 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.500124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.500138 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:00.500187 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.500189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.500233 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.500647 (+ 414us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:00.500748 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.501485 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.502519 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.503072 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.503184 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.503240 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.503428 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.503508 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":54,"threads_started":1}
I20260924 20:20:00.504136 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.499852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48537 (local address 127.18.172.67:39553)
0924 20:20:00.499980 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.499985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.500073 (+ 88us) server_negotiation.cc:407] Connection header received
0924 20:20:00.500170 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.500173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.500221 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.500304 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:00.500579 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.501383 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.502900 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.503489 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.503611 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.503654 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.503948 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.504017 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:00.504145 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.499488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:39553 (local address 127.18.172.66:48537)
0924 20:20:00.499968 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.499982 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.500084 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.500328 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.500331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.500346 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:00.500461 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.500467 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.501492 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.501495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.502775 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.502785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.503722 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.503728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.503818 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.503839 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.503926 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.504006 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
I20260924 20:20:00.504444 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a98250c5d645bf90938f50546868fa" candidate_uuid: "85b69f0d18234e4fa7737000dcc05123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074b785f63034bebbb2cd00c99486fd1" is_pre_election: true
I20260924 20:20:00.504611 20398 raft_consensus.cc:2454] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b69f0d18234e4fa7737000dcc05123 in term 0.
I20260924 20:20:00.504571 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a98250c5d645bf90938f50546868fa" candidate_uuid: "85b69f0d18234e4fa7737000dcc05123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" is_pre_election: true
I20260924 20:20:00.504742 20678 raft_consensus.cc:2454] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b69f0d18234e4fa7737000dcc05123 in term 0.
I20260924 20:20:00.504983 20469 leader_election.cc:302] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b785f63034bebbb2cd00c99486fd1, 85b69f0d18234e4fa7737000dcc05123; no voters:
I20260924 20:20:00.505262 20753 raft_consensus.cc:2781] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:00.505327 20753 raft_consensus.cc:482] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:00.505366 20753 raft_consensus.cc:3037] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.506345 20753 raft_consensus.cc:504] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.506695 20753 leader_election.cc:288] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [CANDIDATE]: Term 1 election: Requested vote from peers 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303), 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553)
I20260924 20:20:00.507063 20398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a98250c5d645bf90938f50546868fa" candidate_uuid: "85b69f0d18234e4fa7737000dcc05123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "074b785f63034bebbb2cd00c99486fd1"
I20260924 20:20:00.507103 20678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a98250c5d645bf90938f50546868fa" candidate_uuid: "85b69f0d18234e4fa7737000dcc05123" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f4fb1bd26a24de49db2ae67c0697c39"
I20260924 20:20:00.507176 20398 raft_consensus.cc:3037] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.507249 20678 raft_consensus.cc:3037] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:00.508093 20398 raft_consensus.cc:2454] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b69f0d18234e4fa7737000dcc05123 in term 1.
I20260924 20:20:00.508179 20678 raft_consensus.cc:2454] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b69f0d18234e4fa7737000dcc05123 in term 1.
I20260924 20:20:00.508376 20469 leader_election.cc:302] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074b785f63034bebbb2cd00c99486fd1, 85b69f0d18234e4fa7737000dcc05123; no voters:
I20260924 20:20:00.508548 20753 raft_consensus.cc:2781] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:00.508714 20753 raft_consensus.cc:686] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [term 1 LEADER]: Becoming Leader. State: Replica: 85b69f0d18234e4fa7737000dcc05123, State: Running, Role: LEADER
I20260924 20:20:00.508978 20753 consensus_queue.cc:237] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } }
I20260924 20:20:00.510154 20792 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: TxnStatusTablet state changed. Reason: New leader 85b69f0d18234e4fa7737000dcc05123. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.510270 20792 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:00.510718 20794 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:20:00.510905 20794 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:20:00.511458 20239 catalog_manager.cc:5495] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:00.539527 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.535761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36439 (local address 127.0.0.1:53016)
0924 20:20:00.536011 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.536026 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.536152 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.536496 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.536499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.536507 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:00.536623 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.536628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.537328 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.537334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.537957 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.537964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.538403 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.538406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.538491 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.538543 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.539238 (+ 695us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.539255 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.539320 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.539400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 20:20:00.539527 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.535873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53016 (local address 127.18.172.66:36439)
0924 20:20:00.536007 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.536012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.536074 (+ 62us) server_negotiation.cc:407] Connection header received
0924 20:20:00.536304 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.536307 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.536373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.536473 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:00.536728 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.537222 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.538079 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.538296 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.538404 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.538691 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.539146 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.539265 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.539339 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.539389 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:20:00.542554 20398 raft_consensus.cc:1261] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Refusing update from remote peer 85b69f0d18234e4fa7737000dcc05123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:00.542822 20678 raft_consensus.cc:1261] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Refusing update from remote peer 85b69f0d18234e4fa7737000dcc05123: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:00.543082 20753 consensus_queue.cc:1035] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:00.543402 20792 consensus_queue.cc:1035] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:00.544667 20754 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: TxnStatusTablet state changed. Reason: New leader 85b69f0d18234e4fa7737000dcc05123. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.544777 20754 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.544958 20755 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: TxnStatusTablet state changed. Reason: New leader 85b69f0d18234e4fa7737000dcc05123. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.545051 20755 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.547552 20753 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.547657 20753 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:00.547772 20753 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.547840 20753 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 85b69f0d18234e4fa7737000dcc05123: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:00.548576 20755 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.548693 20755 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 0f4fb1bd26a24de49db2ae67c0697c39: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.552894 20754 tablet_replica.cc:439] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85b69f0d18234e4fa7737000dcc05123" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "074b785f63034bebbb2cd00c99486fd1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45303 } } peers { permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 } } peers { permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 } } }
I20260924 20:20:00.553020 20754 tablet_replica.cc:442] T 81a98250c5d645bf90938f50546868fa P 074b785f63034bebbb2cd00c99486fd1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:00.560379 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.556018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53676 (local address 127.18.172.65:45303)
0924 20:20:00.556137 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.556142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.556278 (+ 136us) server_negotiation.cc:407] Connection header received
0924 20:20:00.556409 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.556413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.556475 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.556584 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:00.556914 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.557723 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.558696 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.559075 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.559176 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.559434 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.560002 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.560114 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.560181 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.560239 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:00.571506 20742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.567651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33865 (local address 127.18.172.66:36439)
0924 20:20:00.567796 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.567800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.567944 (+ 144us) server_negotiation.cc:407] Connection header received
0924 20:20:00.568068 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.568070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.568119 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.568175 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:00.568612 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.569132 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.570358 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.570567 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.570653 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.571116 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.571184 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.571238 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.571358 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.571402 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:20:00.571611 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.567513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36439 (local address 127.18.172.65:33865)
0924 20:20:00.567852 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.567868 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.567973 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.568209 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.568212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.568221 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:00.568348 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.568355 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.569240 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.569244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.570240 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.570251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.570704 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.570707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.570867 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.570932 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.571266 (+ 334us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.571273 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.571343 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.571423 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":104,"threads_started":1}
I20260924 20:20:00.578809 20538 raft_consensus.cc:1261] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Refusing update from remote peer 074b785f63034bebbb2cd00c99486fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:00.578858 20678 raft_consensus.cc:1261] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Refusing update from remote peer 074b785f63034bebbb2cd00c99486fd1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:00.579758 20777 consensus_queue.cc:1035] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f4fb1bd26a24de49db2ae67c0697c39" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:00.580024 20754 consensus_queue.cc:1035] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85b69f0d18234e4fa7737000dcc05123" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:00.584128 20800 mvcc.cc:205] Tried to move back new op lower bound from 7332991797566341120 to 7332991796537155584. Current Snapshot: MvccSnapshot[applied={T|T < 7332991797566341120}]
I20260924 20:20:00.605224 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.600654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:45069 (local address 127.18.172.65:45303)
0924 20:20:00.600771 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.600776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.601095 (+ 319us) server_negotiation.cc:407] Connection header received
0924 20:20:00.601295 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.601298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.601355 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.601430 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:00.601679 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.602476 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.603558 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.603853 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.603976 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.604784 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.604866 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.604949 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.605034 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.605085 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:20:00.605250 20813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.600606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45303 (local address 127.18.172.66:45069)
0924 20:20:00.600997 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:00.601013 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:00.601130 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:00.601457 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:00.601460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:00.601467 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:00.601573 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.601579 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.602588 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.602591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.603437 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:00.603447 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.604461 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.604465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:00.604585 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.604642 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:00.604953 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:00.604961 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:20:00.605025 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 20:20:00.605108 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
I20260924 20:20:00.629791 20240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:00.629969 20240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33808:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:00.632304 20240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 64c41f3eed234566ba6e707641923d3f: Sending DeleteTablet for 3 replicas of tablet 0c721d3dc4fd4846959a54170015a167
I20260924 20:20:00.632869 20378 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c721d3dc4fd4846959a54170015a167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:53666
I20260924 20:20:00.633039 20658 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c721d3dc4fd4846959a54170015a167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35216
I20260924 20:20:00.633508 20518 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c721d3dc4fd4846959a54170015a167 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:53004
I20260924 20:20:00.633569 20817 tablet_replica.cc:330] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: stopping tablet replica
I20260924 20:20:00.633790 20817 raft_consensus.cc:2229] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:00.634069 20818 tablet_replica.cc:330] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: stopping tablet replica
I20260924 20:20:00.634220 20818 raft_consensus.cc:2229] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:00.634327 20818 raft_consensus.cc:2258] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:00.634549 20819 tablet_replica.cc:330] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: stopping tablet replica
I20260924 20:20:00.634658 20819 raft_consensus.cc:2229] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:00.634763 20819 raft_consensus.cc:2258] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:00.635464 20819 ts_tablet_manager.cc:1905] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:00.635793 20817 raft_consensus.cc:2258] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:00.636483 20817 ts_tablet_manager.cc:1905] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:00.638406 20819 ts_tablet_manager.cc:1918] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:00.638489 20819 log.cc:1196] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/0c721d3dc4fd4846959a54170015a167
I20260924 20:20:00.638777 20819 ts_tablet_manager.cc:1939] T 0c721d3dc4fd4846959a54170015a167 P 85b69f0d18234e4fa7737000dcc05123: Deleting consensus metadata
I20260924 20:20:00.639392 20818 ts_tablet_manager.cc:1905] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:00.640756 20817 ts_tablet_manager.cc:1918] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:00.640883 20817 log.cc:1196] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/0c721d3dc4fd4846959a54170015a167
I20260924 20:20:00.641211 20817 ts_tablet_manager.cc:1939] T 0c721d3dc4fd4846959a54170015a167 P 074b785f63034bebbb2cd00c99486fd1: Deleting consensus metadata
I20260924 20:20:00.641964 20225 catalog_manager.cc:4841] TS 074b785f63034bebbb2cd00c99486fd1 (127.18.172.65:45303): tablet 0c721d3dc4fd4846959a54170015a167 (table test-table [id=27a750078c774549bf7975209b6a97c0]) successfully deleted
I20260924 20:20:00.642374 20818 ts_tablet_manager.cc:1918] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:00.642453 20818 log.cc:1196] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/0c721d3dc4fd4846959a54170015a167
I20260924 20:20:00.642700 20818 ts_tablet_manager.cc:1939] T 0c721d3dc4fd4846959a54170015a167 P 0f4fb1bd26a24de49db2ae67c0697c39: Deleting consensus metadata
I20260924 20:20:00.643569 20228 catalog_manager.cc:4841] TS 85b69f0d18234e4fa7737000dcc05123 (127.18.172.66:36439): tablet 0c721d3dc4fd4846959a54170015a167 (table test-table [id=27a750078c774549bf7975209b6a97c0]) successfully deleted
I20260924 20:20:00.644009 20228 catalog_manager.cc:4841] TS 0f4fb1bd26a24de49db2ae67c0697c39 (127.18.172.67:39553): tablet 0c721d3dc4fd4846959a54170015a167 (table test-table [id=27a750078c774549bf7975209b6a97c0]) successfully deleted
Sep 24 20:20:00 dist-test-slave-m1qq krb5kdc[20196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281199, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:20:00.654721 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.639938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.18.172.65:45303)
0924 20:20:00.640057 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.640061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.642490 (+ 2429us) server_negotiation.cc:407] Connection header received
0924 20:20:00.642843 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.642846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.642895 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.642946 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:00.644000 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.644750 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.645935 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.646286 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.646389 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.650330 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:00.650341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:00.650357 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:00.650387 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:00.652562 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.653099 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.653105 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.653107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.653204 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.653589 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.653592 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.653594 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.653924 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:00.654134 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.654365 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.654536 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 20:20:00.655333 20438 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53680
I20260924 20:20:00.666963 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:00.657476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33832 (local address 127.18.172.126:38925)
0924 20:20:00.657628 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:00.657632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:00.658152 (+ 520us) server_negotiation.cc:407] Connection header received
0924 20:20:00.658243 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:00.658246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:00.658301 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:00.658390 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:00.659275 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.660101 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.661272 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:00.661636 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:00.661754 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:00.663368 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:00.663372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:00.663375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:00.663405 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:00.665132 (+ 1727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.665604 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.665608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.665610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.665719 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:00.666014 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:00.666018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:00.666020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:00.666259 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:00.666401 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:00.666644 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 20:20:00.666788 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 20:20:00.667688 20240 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33832
I20260924 20:20:00.669126 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20313
I20260924 20:20:00.676910 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20453
I20260924 20:20:00.682683 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20593
I20260924 20:20:00.689286 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20209
2026-09-24T20:20:00Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3088 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:00 dist-test-slave-m1qq krb5kdc[20826](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:00 dist-test-slave-m1qq krb5kdc[20826](info): set up 2 sockets
Sep 24 20:20:00 dist-test-slave-m1qq krb5kdc[20826](info): commencing operation
krb5kdc: starting...
W20260924 20:20:01.732070 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:01 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:01.860543 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36533
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:01.973084 20839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:01.973356 20839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:01.973397 20839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:01.977619 20839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:01.977677 20839 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:20:01.977694 20839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:01.977713 20839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:01.977730 20839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:01.977788 20839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:01.983136 20839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:36533
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:01.984236 20839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:01.985335 20839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:01.995816 20844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:01.995873 20845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:01.996130 20847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:01.995988 20839 server_base.cc:1034] running on GCE node
I20260924 20:20:01.996892 20839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:01.998346 20839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:01.999591 20839 hybrid_clock.cc:648] HybridClock initialized: now 1790281201999565 us; error 47 us; skew 500 ppm
Sep 24 20:20:01 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281201, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:02.002578 20839 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:02.003489 20839 webserver.cc:460] Webserver started at http://127.18.172.126:36001/ using document root <none> and password file <none>
I20260924 20:20:02.004005 20839 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:02.004072 20839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:02.004232 20839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:02.006212 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "e1f448be720941f9943cc12224a1c94c"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "b66ac3320b588b38543c19fab567869f"
server_key_iv: "b4a9ee1c3a458eb78c8216ea18be510a"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.006732 20839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "e1f448be720941f9943cc12224a1c94c"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "b66ac3320b588b38543c19fab567869f"
server_key_iv: "b4a9ee1c3a458eb78c8216ea18be510a"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.010478 20839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260924 20:20:02.012979 20854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.014448 20839 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:02.014685 20839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "e1f448be720941f9943cc12224a1c94c"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "b66ac3320b588b38543c19fab567869f"
server_key_iv: "b4a9ee1c3a458eb78c8216ea18be510a"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.014788 20839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:02.027678 20839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:02.030730 20839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:02.030943 20839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:02.038724 20839 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:36533
I20260924 20:20:02.038753 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:36533 every 8 connection(s)
I20260924 20:20:02.039742 20839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:02.042557 20917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.047199 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20839
I20260924 20:20:02.047314 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:02.047596 19121 external_mini_cluster.cc:1428] Setting key 9c40e9182172a1127e1633d09f4dacb5
I20260924 20:20:02.048481 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Bootstrap starting.
I20260924 20:20:02.050876 20917 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.051872 20917 log.cc:824] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:02.054003 20917 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: No bootstrap required, opened a new log
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.058483 20917 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } }
I20260924 20:20:02.058802 20917 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.058874 20917 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f448be720941f9943cc12224a1c94c, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.059401 20917 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } }
I20260924 20:20:02.059578 20917 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:02.059652 20917 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:02.059753 20917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:02.060925 20917 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } }
I20260924 20:20:02.061295 20917 leader_election.cc:302] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1f448be720941f9943cc12224a1c94c; no voters:
I20260924 20:20:02.061654 20917 leader_election.cc:288] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:02.061800 20922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:02.062134 20922 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [term 1 LEADER]: Becoming Leader. State: Replica: e1f448be720941f9943cc12224a1c94c, State: Running, Role: LEADER
I20260924 20:20:02.062508 20922 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } }
I20260924 20:20:02.063059 20917 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:02.063212 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.048977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.18.172.126:36533)
0924 20:20:02.049414 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.049431 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.049467 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:20:02.050192 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.050221 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.050580 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.050871 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.051844 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.052921 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.054095 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.054585 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.054717 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.057678 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.057684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.057702 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.057732 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.059751 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.060236 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.060242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.060247 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.060384 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.060643 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.060646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.060648 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.061188 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.061377 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.061853 (+ 476us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.062170 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":138,"threads_started":1}
I20260924 20:20:02.063781 20923 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1f448be720941f9943cc12224a1c94c. Latest consensus state: current_term: 1 leader_uuid: "e1f448be720941f9943cc12224a1c94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } } }
I20260924 20:20:02.063911 20923 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:02.064203 20924 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1f448be720941f9943cc12224a1c94c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1f448be720941f9943cc12224a1c94c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36533 } } }
I20260924 20:20:02.064291 20924 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:02.064603 20930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:02.067508 20930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:02.073006 20930 catalog_manager.cc:1347] Generated new cluster ID: f73190f074164156b1bfa87d67bc5924
I20260924 20:20:02.073093 20930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:02.097239 20930 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:02.098611 20930 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:02.113451 20930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Generated new TSK 0
I20260924 20:20:02.114331 20930 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:02.146397 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36533
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:02.270203 20943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:02.270520 20943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:02.270576 20943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:02.275204 20943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:02.275272 20943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:02.275313 20943 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:20:02.275327 20943 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:02.275360 20943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:02.280714 20943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:02.281759 20943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:02.282835 20943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:02.292685 20948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:02.292719 20949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.292781 20943 server_base.cc:1034] running on GCE node
W20260924 20:20:02.292938 20951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.293390 20943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:02.293993 20943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:02.295203 20943 hybrid_clock.cc:648] HybridClock initialized: now 1790281202295187 us; error 26 us; skew 500 ppm
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:02.297957 20943 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:02.298848 20943 webserver.cc:460] Webserver started at http://127.18.172.65:34769/ using document root <none> and password file <none>
I20260924 20:20:02.299491 20943 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:02.299585 20943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:02.299814 20943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:02.301827 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "4154ab097046460e8e08a1a782230329"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "8b0b0baff6266fbbe88112ec01a1cbdf"
server_key_iv: "318e8ff08f0ab744b9ed0bfde65ea5e9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.302394 20943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "4154ab097046460e8e08a1a782230329"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "8b0b0baff6266fbbe88112ec01a1cbdf"
server_key_iv: "318e8ff08f0ab744b9ed0bfde65ea5e9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.306157 20943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:02.308629 20958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.309837 20943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:02.309980 20943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "4154ab097046460e8e08a1a782230329"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "8b0b0baff6266fbbe88112ec01a1cbdf"
server_key_iv: "318e8ff08f0ab744b9ed0bfde65ea5e9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.310094 20943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:02.336031 20943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:02.339046 20943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:02.339325 20943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:02.340937 20943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:02.341020 20943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.341090 20943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:02.341137 20943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.352645 20943 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:33459
I20260924 20:20:02.352703 21073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:33459 every 8 connection(s)
I20260924 20:20:02.353762 20943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:02.355470 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.341872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:60361 (local address 127.18.172.126:36533)
0924 20:20:02.342001 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.342005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.343131 (+ 1126us) server_negotiation.cc:407] Connection header received
0924 20:20:02.344314 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.344317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.344379 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.344464 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.345881 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.346400 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.347802 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.348093 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.348212 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.351225 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.351229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.351232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.351269 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.353001 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.353605 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.353609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.353611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.353705 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.354072 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.354075 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.354076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.354320 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.354419 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.355213 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.355340 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:02.356208 20967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.342436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.65:60361)
0924 20:20:02.342929 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.342971 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.344032 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.344656 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.344666 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.345133 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.345709 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.345745 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.346526 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.346529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.347647 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.347656 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.348263 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.348268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.348456 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.349116 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.349143 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.350993 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.353118 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.353123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.353138 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.353476 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.353802 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.353806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.353808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.353967 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.354473 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.354478 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.354778 (+ 300us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.354936 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.355269 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260924 20:20:02.357744 21074 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36533
I20260924 20:20:02.357908 21074 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:02.358475 21074 heartbeater.cc:511] Master 127.18.172.126:36533 requested a full tablet report, sending...
I20260924 20:20:02.360286 20869 ts_manager.cc:194] Registered new tserver with Master: 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459)
I20260924 20:20:02.361799 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:60361
I20260924 20:20:02.363966 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 20943
I20260924 20:20:02.364117 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:02.364488 19121 external_mini_cluster.cc:1428] Setting key a1212185dc0c4591c2ab38c62b8be1f5
I20260924 20:20:02.368568 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.360117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:35983 (local address 127.18.172.126:36533)
0924 20:20:02.360246 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.360250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.360467 (+ 217us) server_negotiation.cc:407] Connection header received
0924 20:20:02.360523 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.360526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.360582 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.360656 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.361296 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.362108 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.363143 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.363446 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.363559 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.365041 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.365048 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.365054 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.365086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.366733 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.367302 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.367304 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.367306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.367380 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.367735 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.367740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.367741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.367920 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.368027 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.368294 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.368403 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:02.368580 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.359967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.65:35983)
0924 20:20:02.360306 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.360324 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.360440 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.360706 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.360709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.360997 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.361124 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.361132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.362270 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.362273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.362949 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.362958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.363575 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.363579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.363794 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.364404 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.364420 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.364878 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.366847 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.366854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.366859 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.367207 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.367522 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.367525 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.367527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.367625 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.368017 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.368020 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.368130 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.368239 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.368403 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:02.394811 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36533
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:02.507615 21083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:02.507879 21083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:02.507926 21083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:02.511979 21083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:02.512035 21083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:02.512075 21083 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:20:02.512089 21083 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:02.512121 21083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:02.517351 21083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:02.518465 21083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:02.519577 21083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:02.529096 21091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:02.529088 21088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.529352 21083 server_base.cc:1034] running on GCE node
W20260924 20:20:02.529090 21089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.529837 21083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:02.530458 21083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:02.531649 21083 hybrid_clock.cc:648] HybridClock initialized: now 1790281202531630 us; error 28 us; skew 500 ppm
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:02.534674 21083 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:02.535629 21083 webserver.cc:460] Webserver started at http://127.18.172.66:45957/ using document root <none> and password file <none>
I20260924 20:20:02.536178 21083 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:02.536240 21083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:02.536404 21083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:02.538483 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "89873710cef64717ae120ba8dc111dfe"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "81d5bb90481f3e86a56553f526ac1468"
server_key_iv: "b022f057ba6f3872ed7f13f461d91e43"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.539091 21083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "89873710cef64717ae120ba8dc111dfe"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "81d5bb90481f3e86a56553f526ac1468"
server_key_iv: "b022f057ba6f3872ed7f13f461d91e43"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.542913 21083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:20:02.545449 21098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.546788 21083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:02.546933 21083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "89873710cef64717ae120ba8dc111dfe"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "81d5bb90481f3e86a56553f526ac1468"
server_key_iv: "b022f057ba6f3872ed7f13f461d91e43"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.547077 21083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:02.569155 21083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:02.572827 21083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:02.573031 21083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:02.574522 21083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:02.574599 21083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.574666 21083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:02.574710 21083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.585534 21083 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36033
I20260924 20:20:02.585592 21213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36033 every 8 connection(s)
I20260924 20:20:02.586412 21083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:02.589669 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.575757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:49181 (local address 127.18.172.126:36533)
0924 20:20:02.575913 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.575920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.576841 (+ 921us) server_negotiation.cc:407] Connection header received
0924 20:20:02.578376 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.578383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.578457 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.578669 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.580445 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.581304 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.582565 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.582780 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.582859 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.586035 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.586038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.586040 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.586064 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.587559 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.588132 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.588136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.588138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.588228 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.588582 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.588587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.588589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.588770 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.588877 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.589350 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.589501 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:20:02.590418 21107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.576187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.66:49181)
0924 20:20:02.576684 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.576744 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.578122 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.578875 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.578887 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.579483 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.580264 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.580288 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.581437 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.581440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.582387 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.582424 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.582946 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.582952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.583215 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.583960 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.583986 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.585863 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.587684 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.587689 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.587703 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.588023 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.588341 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.588345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.588347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.588487 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.588875 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.588882 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.589119 (+ 237us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.589310 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.589627 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
I20260924 20:20:02.591389 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21083
I20260924 20:20:02.591508 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:02.591778 21214 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36533
I20260924 20:20:02.591815 19121 external_mini_cluster.cc:1428] Setting key abff91ba623514ac8f4f79df0c863e42
I20260924 20:20:02.591958 21214 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:02.592571 21214 heartbeater.cc:511] Master 127.18.172.126:36533 requested a full tablet report, sending...
I20260924 20:20:02.593623 20871 ts_manager.cc:194] Registered new tserver with Master: 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:02.594204 20871 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:49181
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:02.603641 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.594714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52421 (local address 127.18.172.126:36533)
0924 20:20:02.596049 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.596052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.596064 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:02.596092 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.596094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.596125 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.596194 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.596593 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.597414 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.598783 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.599054 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.599167 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.600464 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.600467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.600469 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.600490 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.601875 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.602335 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.602337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.602338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.602436 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.602761 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.602763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.602765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.602961 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.603112 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.603436 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.603526 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1272}
I20260924 20:20:02.603724 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.594678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.66:52421)
0924 20:20:02.594960 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.594975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.595119 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.596227 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.596229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.596385 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.596477 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.596482 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.597566 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.597572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.598650 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.598676 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.599188 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.599192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.599332 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.599816 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.599836 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.600329 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.601985 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.601992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.601996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.602242 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.602544 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.602547 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.602549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.602647 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.603111 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.603114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.603217 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.603385 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.603542 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:02.622927 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36533
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:02.732784 21223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:02.733101 21223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:02.733189 21223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:02.737314 21223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:02.737403 21223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:02.737490 21223 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 20:20:02.737537 21223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:02.737601 21223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:02.742830 21223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:36533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:02.743930 21223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:02.744961 21223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:02.754566 21231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:02.754557 21228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.754628 21223 server_base.cc:1034] running on GCE node
W20260924 20:20:02.754554 21229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:02.755336 21223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:02.756016 21223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:02.757215 21223 hybrid_clock.cc:648] HybridClock initialized: now 1790281202757196 us; error 31 us; skew 500 ppm
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:02.760186 21223 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:02.761052 21223 webserver.cc:460] Webserver started at http://127.18.172.67:37221/ using document root <none> and password file <none>
I20260924 20:20:02.761606 21223 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:02.761703 21223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:02.761931 21223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:02.764181 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "32a86dea1a254f08b45c995d39384666"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "982094207133883250475fcb0722241d"
server_key_iv: "b9d7ed3dfd19258c1bd21da8856004d9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.764732 21223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "32a86dea1a254f08b45c995d39384666"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "982094207133883250475fcb0722241d"
server_key_iv: "b9d7ed3dfd19258c1bd21da8856004d9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.768419 21223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:02.770804 21238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.772163 21223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:02.772313 21223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "32a86dea1a254f08b45c995d39384666"
format_stamp: "Formatted at 2026-09-24 20:20:02 on dist-test-slave-m1qq"
server_key: "982094207133883250475fcb0722241d"
server_key_iv: "b9d7ed3dfd19258c1bd21da8856004d9"
server_key_version: "encryptionkey@0"
I20260924 20:20:02.772445 21223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:02.798301 21223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:02.801704 21223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:02.801909 21223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:02.803578 21223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:02.803654 21223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:02.803730 21223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:02.803781 21223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.814770 21223 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:37741
I20260924 20:20:02.814966 21353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:37741 every 8 connection(s)
I20260924 20:20:02.815726 21223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:02.818198 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.804667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:40439 (local address 127.18.172.126:36533)
0924 20:20:02.804812 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.804816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.805800 (+ 984us) server_negotiation.cc:407] Connection header received
0924 20:20:02.806903 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.806906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.806964 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.807083 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.808796 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.809408 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.810566 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.810848 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.810971 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.814268 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.814272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.814276 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.814305 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.816037 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.816638 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.816643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.816646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.816737 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.817079 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.817082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.817085 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.817273 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.817383 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.817911 (+ 528us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.818031 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:20:02.819064 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.805146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.67:40439)
0924 20:20:02.805644 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.805688 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.806659 (+ 971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.807294 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.807306 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.807798 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.808560 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.808583 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.809549 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.809552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.810437 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.810446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.811045 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.811049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.811300 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.811981 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.812008 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.814065 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.816152 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.816158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.816173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.816517 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.816838 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.816840 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.816843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.816982 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.817379 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.817385 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.817664 (+ 279us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.817855 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.818172 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":133,"threads_started":1}
I20260924 20:20:02.819089 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21223
I20260924 20:20:02.819197 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:02.819499 19121 external_mini_cluster.cc:1428] Setting key b20abe0a5b19a2187a6d75e12d080e37
I20260924 20:20:02.820389 21354 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36533
I20260924 20:20:02.820528 21354 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:02.821094 21354 heartbeater.cc:511] Master 127.18.172.126:36533 requested a full tablet report, sending...
I20260924 20:20:02.822157 20869 ts_manager.cc:194] Registered new tserver with Master: 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741)
I20260924 20:20:02.822878 20869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:40439
I20260924 20:20:02.829424 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.822423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:35863 (local address 127.18.172.126:36533)
0924 20:20:02.822542 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.822546 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.822780 (+ 234us) server_negotiation.cc:407] Connection header received
0924 20:20:02.822894 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.822897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.822938 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.823045 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.823548 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.824063 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.824897 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.825129 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.825232 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.826338 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.826341 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.826344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.826369 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.827856 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.828285 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.828288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.828290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.828368 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.828649 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.828653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.828655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.828819 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.828922 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.829178 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.829280 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:20:02.829435 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.822444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.18.172.67:35863)
0924 20:20:02.822700 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.822712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.822818 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.823090 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.823093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.823317 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.823436 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.823443 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.824198 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.824201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.824778 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.824786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.825247 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.825250 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.825346 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.825882 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.825896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.826198 (+ 302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.827984 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.827988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.827990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.828185 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.828465 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.828468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.828470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.828553 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.828922 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.828924 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.829018 (+ 94us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.829124 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.829260 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260924 20:20:02.833104 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.858196 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.849652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.18.172.126:36533)
0924 20:20:02.849795 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.849798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.849810 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:02.849850 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.849852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.849889 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.849982 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.850743 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.851388 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.852230 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.852458 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.852572 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.855055 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.855059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.855061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.855087 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.856377 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.856866 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.856871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.856873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.856956 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.857230 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.857234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.857236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.857458 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.857613 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.857886 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.858017 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:20:02.861349 20869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:02.864615 20869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:02.880370 21373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.873535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33459 (local address 127.0.0.1:59638)
0924 20:20:02.873924 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.873977 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.874131 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.875249 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.875255 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.875279 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:02.875428 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.875436 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.877186 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.877212 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.878638 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.878648 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.879777 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.879781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.879928 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.879980 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.880070 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.880199 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":62,"thread_start_us":104,"threads_started":1}
I20260924 20:20:02.881069 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.874055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.18.172.65:33459)
0924 20:20:02.874709 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.874717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.874742 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:20:02.874819 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.874826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.875077 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.875622 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:02.875754 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.877000 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.878769 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.879612 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.880571 (+ 959us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.880733 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.880803 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.880905 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":124,"threads_started":1}
I20260924 20:20:02.881675 21374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.874087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36033 (local address 127.0.0.1:35260)
0924 20:20:02.874372 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.874385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.874465 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.875916 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.875919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.875937 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:02.876080 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.876088 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.878112 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.878116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.879650 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.879658 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.881109 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.881112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.881328 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.881353 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.881443 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.881522 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"mutex_wait_us":7,"thread_start_us":97,"threads_started":1}
I20260924 20:20:02.882216 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.874705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.18.172.66:36033)
0924 20:20:02.875302 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.875313 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.875334 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:20:02.875455 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.875464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.875738 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.876283 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:02.876454 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.877938 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.879782 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.880778 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.880881 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.881051 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.881959 (+ 908us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.882056 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":145,"threads_started":1}
I20260924 20:20:02.882606 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.873145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37741 (local address 127.0.0.1:55332)
0924 20:20:02.875360 (+ 2215us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.875376 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.875576 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.876320 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.876324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.876348 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:02.876466 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.876474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.879657 (+ 3183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.879660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.880926 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.880937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.882235 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.882239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.882333 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.882355 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.882418 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.882481 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260924 20:20:02.882828 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.873271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.18.172.67:37741)
0924 20:20:02.875268 (+ 1997us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.875282 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.875499 (+ 217us) server_negotiation.cc:407] Connection header received
0924 20:20:02.875747 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.875754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.876009 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.876156 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:02.876713 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.879499 (+ 2786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.881100 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.882105 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.882219 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.882472 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.882524 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.882608 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1885,"thread_start_us":99,"threads_started":1}
I20260924 20:20:02.884526 21008 tablet_service.cc:1467] Processing CreateTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d (DEFAULT_TABLE table=test-table [id=abdc141d01ca426ab57c1397c47801ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:02.884940 21148 tablet_service.cc:1467] Processing CreateTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d (DEFAULT_TABLE table=test-table [id=abdc141d01ca426ab57c1397c47801ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:02.884938 21288 tablet_service.cc:1467] Processing CreateTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d (DEFAULT_TABLE table=test-table [id=abdc141d01ca426ab57c1397c47801ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:02.885681 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c6224ef1f640faadaa8cf6d1e26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.885895 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c6224ef1f640faadaa8cf6d1e26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.886013 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8c6224ef1f640faadaa8cf6d1e26b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.892752 21377 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Bootstrap starting.
I20260924 20:20:02.893205 21378 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Bootstrap starting.
I20260924 20:20:02.894155 21379 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Bootstrap starting.
I20260924 20:20:02.895646 21378 tablet_bootstrap.cc:654] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.896257 21379 tablet_bootstrap.cc:654] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.896543 21378 log.cc:824] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:02.896956 21377 tablet_bootstrap.cc:654] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.897075 21379 log.cc:824] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:02.897812 21377 log.cc:824] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:02.898607 21378 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: No bootstrap required, opened a new log
I20260924 20:20:02.898803 21378 ts_tablet_manager.cc:1397] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:20:02.899102 21379 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: No bootstrap required, opened a new log
I20260924 20:20:02.899273 21379 ts_tablet_manager.cc:1397] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 20:20:02.899681 21377 tablet_bootstrap.cc:492] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: No bootstrap required, opened a new log
I20260924 20:20:02.899865 21377 ts_tablet_manager.cc:1397] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:20:02.902706 21377 raft_consensus.cc:348] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.902983 21377 raft_consensus.cc:374] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.903059 21377 raft_consensus.cc:729] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.903302 21379 raft_consensus.cc:348] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.903360 21378 raft_consensus.cc:348] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.903563 21379 raft_consensus.cc:374] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.903563 21378 raft_consensus.cc:374] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.903620 21378 raft_consensus.cc:729] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154ab097046460e8e08a1a782230329, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.903620 21379 raft_consensus.cc:729] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89873710cef64717ae120ba8dc111dfe, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.903563 21377 consensus_queue.cc:260] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.904040 21379 consensus_queue.cc:260] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.904040 21378 consensus_queue.cc:260] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.904241 21354 heartbeater.cc:503] Master 127.18.172.126:36533 was elected leader, sending a full tablet report...
I20260924 20:20:02.904628 21377 ts_tablet_manager.cc:1428] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260924 20:20:02.904965 21379 ts_tablet_manager.cc:1428] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:20:02.904965 21378 ts_tablet_manager.cc:1428] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260924 20:20:02.905453 21214 heartbeater.cc:503] Master 127.18.172.126:36533 was elected leader, sending a full tablet report...
I20260924 20:20:02.906271 21074 heartbeater.cc:503] Master 127.18.172.126:36533 was elected leader, sending a full tablet report...
I20260924 20:20:02.915516 21383 raft_consensus.cc:482] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:02.915730 21383 raft_consensus.cc:504] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.916800 21383 leader_election.cc:288] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033), 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459)
I20260924 20:20:02.921461 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.916978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36033 (local address 127.18.172.67:50107)
0924 20:20:02.917091 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.917109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.917210 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.917525 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.917529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.917551 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:02.917683 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.917689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.918739 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.918742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.920113 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.920124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.920992 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.920994 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.921110 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.921130 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.921211 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.921300 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 20:20:02.921494 21376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.917172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:50107 (local address 127.18.172.66:36033)
0924 20:20:02.917290 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.917294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.917310 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:02.917366 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.917368 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.917404 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.917466 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:02.917812 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.918635 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.920267 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.920872 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.920986 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.921038 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.921243 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.921338 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:20:02.921664 21386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.917058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33459 (local address 127.18.172.67:45939)
0924 20:20:02.917438 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.917472 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.917569 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.917763 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.917769 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.917789 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:02.917940 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.917946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.918851 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.918853 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.920440 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.920447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.921277 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.921280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.921364 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.921384 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.921464 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.921530 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":129,"threads_started":1}
I20260924 20:20:02.921677 21375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.917184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:45939 (local address 127.18.172.65:33459)
0924 20:20:02.917290 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.917294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.917576 (+ 282us) server_negotiation.cc:407] Connection header received
0924 20:20:02.917633 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.917635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.917673 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.917729 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:02.918058 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.918766 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.920542 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.921152 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.921299 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.921343 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.921490 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.921568 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:20:02.922294 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c6224ef1f640faadaa8cf6d1e26b9d" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329" is_pre_election: true
I20260924 20:20:02.922326 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c6224ef1f640faadaa8cf6d1e26b9d" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe" is_pre_election: true
I20260924 20:20:02.922592 21028 raft_consensus.cc:2454] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:02.922770 21168 raft_consensus.cc:2454] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:02.923254 21241 leader_election.cc:302] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329; no voters:
I20260924 20:20:02.923564 21383 raft_consensus.cc:2781] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:02.923692 21383 raft_consensus.cc:482] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:02.923754 21383 raft_consensus.cc:3037] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:02.924763 21383 raft_consensus.cc:504] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.925110 21383 leader_election.cc:288] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Requested vote from peers 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033), 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459)
I20260924 20:20:02.925433 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c6224ef1f640faadaa8cf6d1e26b9d" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe"
I20260924 20:20:02.925565 21168 raft_consensus.cc:3037] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:02.925596 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8c6224ef1f640faadaa8cf6d1e26b9d" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329"
I20260924 20:20:02.925733 21028 raft_consensus.cc:3037] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:02.926628 21168 raft_consensus.cc:2454] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:02.926853 21028 raft_consensus.cc:2454] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:02.926980 21242 leader_election.cc:302] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 89873710cef64717ae120ba8dc111dfe; no voters:
I20260924 20:20:02.927192 21383 raft_consensus.cc:2781] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:02.927426 21383 raft_consensus.cc:686] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Becoming Leader. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Running, Role: LEADER
I20260924 20:20:02.927779 21383 consensus_queue.cc:237] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } }
I20260924 20:20:02.930830 20870 catalog_manager.cc:5495] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a86dea1a254f08b45c995d39384666 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:20:02 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281201, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:02.973650 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.964984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43408 (local address 127.18.172.126:36533)
0924 20:20:02.965154 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:02.965161 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:02.965282 (+ 121us) server_negotiation.cc:407] Connection header received
0924 20:20:02.965443 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:02.965446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:02.965488 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:02.965572 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:02.966209 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.966722 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.967569 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.967782 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.967882 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.970173 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:02.970176 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:02.970178 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:02.970197 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:02.971912 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.972390 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.972394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.972396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.972471 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:02.972739 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:02.972742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:02.972744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:02.972909 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:02.973010 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:02.973344 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 20:20:02.973507 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":51}
I20260924 20:20:02.973650 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:02.964897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36533 (local address 127.0.0.1:43408)
0924 20:20:02.965191 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:02.965205 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:02.965335 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:02.965590 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:02.965592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:02.965986 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:02.966078 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.966084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.966828 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.966833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.967454 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:02.967462 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:02.967898 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:02.967901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:02.967976 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:02.968605 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:02.968617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:02.970049 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:02.972041 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.972053 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.972059 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.972289 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.972563 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:02.972566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:02.972567 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:02.972648 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:02.973020 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:02.973028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:02.973151 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 20:20:02.973295 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 20:20:02.973451 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":108,"threads_started":1}
I20260924 20:20:02.977701 20870 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:43408:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:20:02.978428 20870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:02.983808 21288 tablet_service.cc:1467] Processing CreateTablet for tablet 1783ccb8d05c470eaf367de32744773c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8609c26ae6ec4ed0980368d3da181b4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:02.983649 21008 tablet_service.cc:1467] Processing CreateTablet for tablet 1783ccb8d05c470eaf367de32744773c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8609c26ae6ec4ed0980368d3da181b4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:02.984403 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1783ccb8d05c470eaf367de32744773c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.984403 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1783ccb8d05c470eaf367de32744773c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.984037 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 1783ccb8d05c470eaf367de32744773c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8609c26ae6ec4ed0980368d3da181b4d]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:02.984826 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1783ccb8d05c470eaf367de32744773c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:02.987399 21378 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: Bootstrap starting.
I20260924 20:20:02.987488 21377 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: Bootstrap starting.
I20260924 20:20:02.988453 21377 tablet_bootstrap.cc:654] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.988683 21378 tablet_bootstrap.cc:654] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.989672 21377 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: No bootstrap required, opened a new log
I20260924 20:20:02.989776 21377 ts_tablet_manager.cc:1397] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 20:20:02.990240 21377 raft_consensus.cc:348] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.990361 21377 raft_consensus.cc:374] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.990406 21377 raft_consensus.cc:729] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.990401 21378 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: No bootstrap required, opened a new log
I20260924 20:20:02.990489 21378 ts_tablet_manager.cc:1397] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 20:20:02.990536 21377 consensus_queue.cc:260] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.990785 21377 ts_tablet_manager.cc:1428] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260924 20:20:02.991080 21383 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:02.991073 21378 raft_consensus.cc:348] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.991202 21383 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:02.991214 21378 raft_consensus.cc:374] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.991254 21378 raft_consensus.cc:729] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154ab097046460e8e08a1a782230329, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.991484 21378 consensus_queue.cc:260] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.991786 21384 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:02.991940 21384 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:02.993067 21378 ts_tablet_manager.cc:1428] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:02.994437 21379 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: Bootstrap starting.
I20260924 20:20:02.995767 21379 tablet_bootstrap.cc:654] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:02.997064 21379 tablet_bootstrap.cc:492] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: No bootstrap required, opened a new log
I20260924 20:20:02.997180 21379 ts_tablet_manager.cc:1397] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:02.997565 21379 raft_consensus.cc:348] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.997689 21379 raft_consensus.cc:374] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:02.997735 21379 raft_consensus.cc:729] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89873710cef64717ae120ba8dc111dfe, State: Initialized, Role: FOLLOWER
I20260924 20:20:02.997861 21379 consensus_queue.cc:260] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:02.998167 21379 ts_tablet_manager.cc:1428] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:20:02.998152 21385 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:02.998327 21385 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:03.017690 21383 raft_consensus.cc:482] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:03.017846 21383 raft_consensus.cc:504] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.018411 21383 leader_election.cc:288] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:03.018784 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1783ccb8d05c470eaf367de32744773c" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329" is_pre_election: true
I20260924 20:20:03.018790 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1783ccb8d05c470eaf367de32744773c" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe" is_pre_election: true
I20260924 20:20:03.018934 21028 raft_consensus.cc:2454] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:03.018934 21168 raft_consensus.cc:2454] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:03.019277 21242 leader_election.cc:302] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329, 89873710cef64717ae120ba8dc111dfe; no voters:
I20260924 20:20:03.019500 21383 raft_consensus.cc:2781] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:03.019582 21383 raft_consensus.cc:482] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:03.019644 21383 raft_consensus.cc:3037] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.020644 21383 raft_consensus.cc:504] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.020994 21383 leader_election.cc:288] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Requested vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:03.021396 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1783ccb8d05c470eaf367de32744773c" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe"
I20260924 20:20:03.021384 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1783ccb8d05c470eaf367de32744773c" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329"
I20260924 20:20:03.021520 21168 raft_consensus.cc:3037] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.021533 21028 raft_consensus.cc:3037] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.022383 21168 raft_consensus.cc:2454] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:03.022651 21242 leader_election.cc:302] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 89873710cef64717ae120ba8dc111dfe; no voters:
I20260924 20:20:03.022836 21383 raft_consensus.cc:2781] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:03.022930 21383 raft_consensus.cc:686] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Becoming Leader. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Running, Role: LEADER
I20260924 20:20:03.023095 21383 consensus_queue.cc:237] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.023180 21028 raft_consensus.cc:2454] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:03.024152 21387 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: TxnStatusTablet state changed. Reason: New leader 32a86dea1a254f08b45c995d39384666. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.024322 21387 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:03.024688 21399 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:20:03.024917 21399 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:20:03.025907 20870 catalog_manager.cc:5495] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a86dea1a254f08b45c995d39384666 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } health_report { overall_health: HEALTHY } } }
I20260924 20:20:03.050112 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:03.045071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37741 (local address 127.0.0.1:55340)
0924 20:20:03.045354 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:03.045370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:03.045475 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:03.045934 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:03.045937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:03.045944 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:03.046054 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:03.046060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.047173 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.047177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:03.048152 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:03.048163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.048804 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.048807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:03.048896 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:03.048946 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:03.049809 (+ 863us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:03.049823 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 20:20:03.049884 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 20:20:03.049967 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260924 20:20:03.050688 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:03.045177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55340 (local address 127.18.172.67:37741)
0924 20:20:03.045312 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:03.045319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:03.045584 (+ 265us) server_negotiation.cc:407] Connection header received
0924 20:20:03.045654 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:03.045657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:03.045729 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:03.045812 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:03.046185 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.046958 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.048298 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.048633 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.048721 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:03.049097 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:03.049595 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:03.049714 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:03.050512 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 20:20:03.050570 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 20:20:03.056646 21028 raft_consensus.cc:1261] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:03.056713 21168 raft_consensus.cc:1261] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:03.057546 21387 consensus_queue.cc:1035] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:03.062546 21384 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: TxnStatusTablet state changed. Reason: New leader 32a86dea1a254f08b45c995d39384666. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.063532 21384 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:03.064922 21387 consensus_queue.cc:1035] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:20:03.066787 21355 tablet.cc:2369] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:03.067687 21387 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.067829 21387 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:03.067945 21402 mvcc.cc:205] Tried to move back new op lower bound from 7332991807711055872 to 7332991807585173504. Current Snapshot: MvccSnapshot[applied={T|T < 7332991807711055872}]
I20260924 20:20:03.071130 21384 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.071246 21384 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 4154ab097046460e8e08a1a782230329: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:03.072314 21403 mvcc.cc:205] Tried to move back new op lower bound from 7332991807711055872 to 7332991807585173504. Current Snapshot: MvccSnapshot[applied={T|T < 7332991807711055872}]
I20260924 20:20:03.073205 21385 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: TxnStatusTablet state changed. Reason: New leader 32a86dea1a254f08b45c995d39384666. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.073371 21385 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:03.081724 21383 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.081876 21383 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 32a86dea1a254f08b45c995d39384666: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:03.082782 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:03.078357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.18.172.67:37741)
0924 20:20:03.078478 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:03.078482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:03.078551 (+ 69us) server_negotiation.cc:407] Connection header received
0924 20:20:03.078729 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:03.078733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:03.078790 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:03.078882 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:03.079201 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.079978 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.081142 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.081464 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.081581 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:03.081975 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:03.082068 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:03.082137 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:03.082593 (+ 456us) server_negotiation.cc:299] Negotiation successful
0924 20:20:03.082651 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:20:03.083679 21406 mvcc.cc:205] Tried to move back new op lower bound from 7332991807711055872 to 7332991807585173504. Current Snapshot: MvccSnapshot[applied={T|T < 7332991807711055872}]
I20260924 20:20:03.084390 21385 tablet_replica.cc:439] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } } }
I20260924 20:20:03.084515 21385 tablet_replica.cc:442] T 1783ccb8d05c470eaf367de32744773c P 89873710cef64717ae120ba8dc111dfe: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 20:20:03.088472 21215 tablet.cc:2369] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:03.096859 21372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:03.093296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:48455 (local address 127.18.172.67:37741)
0924 20:20:03.093693 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:03.093695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:03.093704 (+ 9us) server_negotiation.cc:407] Connection header received
0924 20:20:03.093730 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:03.093732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:03.093772 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:03.093856 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:03.094160 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.094715 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.095710 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.095939 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.096052 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:03.096362 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:03.096429 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:03.096520 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:03.096597 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 20:20:03.096689 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":349}
I20260924 20:20:03.096859 21360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:03.093256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37741 (local address 127.18.172.67:48455)
0924 20:20:03.093489 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:03.093512 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:03.093588 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:03.093875 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:03.093878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:03.093885 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:03.094054 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:03.094060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.094828 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.094835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:03.095587 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:03.095596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:03.096051 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:03.096054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:03.096147 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:03.096191 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:03.096514 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:03.096519 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:20:03.096591 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 20:20:03.096712 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":43}
I20260924 20:20:03.105382 21028 raft_consensus.cc:1261] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:03.105454 21168 raft_consensus.cc:1261] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 20:20:03.105399 21075 tablet.cc:2369] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:03.105823 21387 consensus_queue.cc:1035] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:03.107154 21387 consensus_queue.cc:1035] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:03.155845 20871 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:03.156059 20871 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:03.158746 20871 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Sending DeleteTablet for 3 replicas of tablet d8c6224ef1f640faadaa8cf6d1e26b9d
I20260924 20:20:03.159559 21288 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:55332
I20260924 20:20:03.159773 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59638
I20260924 20:20:03.160265 21418 tablet_replica.cc:330] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: stopping tablet replica
I20260924 20:20:03.160617 21418 raft_consensus.cc:2229] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:03.160686 21148 tablet_service.cc:1514] Processing DeleteTablet for tablet d8c6224ef1f640faadaa8cf6d1e26b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 20:20:03.160842 21418 raft_consensus.cc:2258] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:03.161880 21419 tablet_replica.cc:330] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: stopping tablet replica
I20260924 20:20:03.161880 20871 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:03.162354 20871 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:03.162493 21419 raft_consensus.cc:2229] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:03.162811 21419 raft_consensus.cc:2258] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:03.163226 21417 tablet_replica.cc:330] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: stopping tablet replica
I20260924 20:20:03.163621 21417 raft_consensus.cc:2229] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:03.164048 21417 raft_consensus.cc:2258] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:03.166443 21417 ts_tablet_manager.cc:1905] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:03.166443 21419 ts_tablet_manager.cc:1905] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:03.168723 21008 tablet_service.cc:1467] Processing CreateTablet for tablet fa25e27b54af42529540c171cf30434a (DEFAULT_TABLE table=test-table [id=bc852e8b378843d392d655856952dfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:03.169070 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa25e27b54af42529540c171cf30434a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:03.169833 21419 ts_tablet_manager.cc:1918] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:03.169942 21419 log.cc:1196] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/d8c6224ef1f640faadaa8cf6d1e26b9d
I20260924 20:20:03.170289 21419 ts_tablet_manager.cc:1939] T d8c6224ef1f640faadaa8cf6d1e26b9d P 89873710cef64717ae120ba8dc111dfe: Deleting consensus metadata
I20260924 20:20:03.170840 21417 ts_tablet_manager.cc:1918] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:03.171039 21417 log.cc:1196] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/d8c6224ef1f640faadaa8cf6d1e26b9d
I20260924 20:20:03.171439 21417 ts_tablet_manager.cc:1939] T d8c6224ef1f640faadaa8cf6d1e26b9d P 32a86dea1a254f08b45c995d39384666: Deleting consensus metadata
I20260924 20:20:03.172413 21418 ts_tablet_manager.cc:1905] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:03.172405 20858 catalog_manager.cc:4841] TS 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033): tablet d8c6224ef1f640faadaa8cf6d1e26b9d (table test-table [id=abdc141d01ca426ab57c1397c47801ef]) successfully deleted
I20260924 20:20:03.172683 20856 catalog_manager.cc:4841] TS 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741): tablet d8c6224ef1f640faadaa8cf6d1e26b9d (table test-table [id=abdc141d01ca426ab57c1397c47801ef]) successfully deleted
I20260924 20:20:03.173381 21148 tablet_service.cc:1467] Processing CreateTablet for tablet fa25e27b54af42529540c171cf30434a (DEFAULT_TABLE table=test-table [id=bc852e8b378843d392d655856952dfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:03.173573 21378 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Bootstrap starting.
I20260924 20:20:03.173668 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa25e27b54af42529540c171cf30434a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:03.174671 21378 tablet_bootstrap.cc:654] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:03.175738 21418 ts_tablet_manager.cc:1918] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:03.175837 21418 log.cc:1196] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/d8c6224ef1f640faadaa8cf6d1e26b9d
I20260924 20:20:03.175828 21288 tablet_service.cc:1467] Processing CreateTablet for tablet fa25e27b54af42529540c171cf30434a (DEFAULT_TABLE table=test-table [id=bc852e8b378843d392d655856952dfde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:03.176110 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa25e27b54af42529540c171cf30434a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:03.176292 21418 ts_tablet_manager.cc:1939] T d8c6224ef1f640faadaa8cf6d1e26b9d P 4154ab097046460e8e08a1a782230329: Deleting consensus metadata
I20260924 20:20:03.176576 21378 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: No bootstrap required, opened a new log
I20260924 20:20:03.176661 21378 ts_tablet_manager.cc:1397] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:03.177068 21378 raft_consensus.cc:348] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.177168 21378 raft_consensus.cc:374] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:03.177198 21378 raft_consensus.cc:729] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154ab097046460e8e08a1a782230329, State: Initialized, Role: FOLLOWER
I20260924 20:20:03.177310 21378 consensus_queue.cc:260] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.177591 21378 ts_tablet_manager.cc:1428] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:03.177616 20857 catalog_manager.cc:4841] TS 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459): tablet d8c6224ef1f640faadaa8cf6d1e26b9d (table test-table [id=abdc141d01ca426ab57c1397c47801ef]) successfully deleted
I20260924 20:20:03.178714 21377 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Bootstrap starting.
I20260924 20:20:03.179999 21377 tablet_bootstrap.cc:654] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:03.180104 21379 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Bootstrap starting.
I20260924 20:20:03.181124 21379 tablet_bootstrap.cc:654] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:03.181248 21377 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: No bootstrap required, opened a new log
I20260924 20:20:03.181360 21377 ts_tablet_manager.cc:1397] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:03.181886 21377 raft_consensus.cc:348] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.182058 21377 raft_consensus.cc:374] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:03.182106 21377 raft_consensus.cc:729] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Initialized, Role: FOLLOWER
I20260924 20:20:03.182331 21379 tablet_bootstrap.cc:492] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: No bootstrap required, opened a new log
I20260924 20:20:03.182420 21379 ts_tablet_manager.cc:1397] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:03.182338 21377 consensus_queue.cc:260] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.182669 21377 ts_tablet_manager.cc:1428] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:03.182829 21379 raft_consensus.cc:348] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.182940 21379 raft_consensus.cc:374] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:03.182971 21379 raft_consensus.cc:729] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89873710cef64717ae120ba8dc111dfe, State: Initialized, Role: FOLLOWER
I20260924 20:20:03.183113 21379 consensus_queue.cc:260] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.183365 21379 ts_tablet_manager.cc:1428] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:03.253692 21383 raft_consensus.cc:482] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:03.253832 21383 raft_consensus.cc:504] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.254343 21383 leader_election.cc:288] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:03.254771 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa25e27b54af42529540c171cf30434a" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329" is_pre_election: true
I20260924 20:20:03.254802 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa25e27b54af42529540c171cf30434a" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe" is_pre_election: true
I20260924 20:20:03.254920 21027 raft_consensus.cc:2454] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:03.254944 21168 raft_consensus.cc:2454] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:03.255252 21241 leader_election.cc:302] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329; no voters:
I20260924 20:20:03.255414 21383 raft_consensus.cc:2781] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:03.255478 21383 raft_consensus.cc:482] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:03.255545 21383 raft_consensus.cc:3037] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.256484 21383 raft_consensus.cc:504] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.257009 21383 leader_election.cc:288] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Requested vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:03.257351 21168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa25e27b54af42529540c171cf30434a" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe"
I20260924 20:20:03.257354 21027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa25e27b54af42529540c171cf30434a" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329"
I20260924 20:20:03.257490 21027 raft_consensus.cc:3037] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.257519 21168 raft_consensus.cc:3037] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:03.258543 21168 raft_consensus.cc:2454] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:03.258683 21027 raft_consensus.cc:2454] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:03.258896 21242 leader_election.cc:302] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 89873710cef64717ae120ba8dc111dfe; no voters:
I20260924 20:20:03.259172 21383 raft_consensus.cc:2781] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:03.259275 21383 raft_consensus.cc:686] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Becoming Leader. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Running, Role: LEADER
I20260924 20:20:03.259419 21383 consensus_queue.cc:237] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:03.261266 20871 catalog_manager.cc:5495] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a86dea1a254f08b45c995d39384666 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } health_report { overall_health: HEALTHY } } }
I20260924 20:20:03.290150 21168 raft_consensus.cc:1261] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:03.290544 21027 raft_consensus.cc:1261] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:03.290578 21383 consensus_queue.cc:1035] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:03.290920 21383 consensus_queue.cc:1035] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:20:03.305331 21402 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:20:04.859197 20870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:04.859373 20870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43394:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:04.861763 20870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Sending DeleteTablet for 3 replicas of tablet fa25e27b54af42529540c171cf30434a
I20260924 20:20:04.862350 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet fa25e27b54af42529540c171cf30434a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59638
I20260924 20:20:04.862396 21288 tablet_service.cc:1514] Processing DeleteTablet for tablet fa25e27b54af42529540c171cf30434a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:55332
I20260924 20:20:04.862710 21447 tablet_replica.cc:330] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: stopping tablet replica
I20260924 20:20:04.862695 21148 tablet_service.cc:1514] Processing DeleteTablet for tablet fa25e27b54af42529540c171cf30434a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 20:20:04.862797 21448 tablet_replica.cc:330] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: stopping tablet replica
I20260924 20:20:04.862852 21447 raft_consensus.cc:2229] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:04.862923 21448 raft_consensus.cc:2229] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:04.863042 21447 raft_consensus.cc:2258] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:04.863173 21449 tablet_replica.cc:330] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: stopping tablet replica
I20260924 20:20:04.863194 21448 raft_consensus.cc:2258] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:04.863373 21449 raft_consensus.cc:2229] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:04.863559 21449 raft_consensus.cc:2258] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:04.863874 21448 ts_tablet_manager.cc:1905] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:04.864140 21447 ts_tablet_manager.cc:1905] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:04.864360 21449 ts_tablet_manager.cc:1905] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:04.866783 21448 ts_tablet_manager.cc:1918] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:20:04.866782 21449 ts_tablet_manager.cc:1918] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:20:04.866864 21448 log.cc:1196] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/fa25e27b54af42529540c171cf30434a
I20260924 20:20:04.866968 21449 log.cc:1196] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/fa25e27b54af42529540c171cf30434a
I20260924 20:20:04.867199 21448 ts_tablet_manager.cc:1939] T fa25e27b54af42529540c171cf30434a P 32a86dea1a254f08b45c995d39384666: Deleting consensus metadata
I20260924 20:20:04.867331 21449 ts_tablet_manager.cc:1939] T fa25e27b54af42529540c171cf30434a P 89873710cef64717ae120ba8dc111dfe: Deleting consensus metadata
I20260924 20:20:04.867363 21447 ts_tablet_manager.cc:1918] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:20:04.867445 21447 log.cc:1196] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/fa25e27b54af42529540c171cf30434a
I20260924 20:20:04.867772 21447 ts_tablet_manager.cc:1939] T fa25e27b54af42529540c171cf30434a P 4154ab097046460e8e08a1a782230329: Deleting consensus metadata
I20260924 20:20:04.867959 20856 catalog_manager.cc:4841] TS 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741): tablet fa25e27b54af42529540c171cf30434a (table test-table [id=bc852e8b378843d392d655856952dfde]) successfully deleted
I20260924 20:20:04.868172 20858 catalog_manager.cc:4841] TS 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033): tablet fa25e27b54af42529540c171cf30434a (table test-table [id=bc852e8b378843d392d655856952dfde]) successfully deleted
I20260924 20:20:04.868432 20857 catalog_manager.cc:4841] TS 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459): tablet fa25e27b54af42529540c171cf30434a (table test-table [id=bc852e8b378843d392d655856952dfde]) successfully deleted
Sep 24 20:20:04 dist-test-slave-m1qq krb5kdc[20826](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:20:04 dist-test-slave-m1qq krb5kdc[20826](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281204, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:04.900580 21457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:04.886892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.18.172.126:36533)
0924 20:20:04.887288 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:04.887295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:04.887312 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:20:04.887355 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:04.887358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:04.887400 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:04.887483 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:04.888260 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:04.888800 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:04.889788 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:04.890006 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:04.890080 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:04.897326 (+ 7246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:04.897329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:04.897331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:04.897360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:04.898965 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:04.899451 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:04.899454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:04.899455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:04.899532 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:04.899821 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:04.899824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:04.899825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:04.900000 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:04.900102 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:04.900299 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 20:20:04.900399 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":212,"threads_started":1}
I20260924 20:20:04.903476 20870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:04.903878 20870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:04.908697 21008 tablet_service.cc:1467] Processing CreateTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e (DEFAULT_TABLE table=test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:04.908866 21148 tablet_service.cc:1467] Processing CreateTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e (DEFAULT_TABLE table=test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:04.908880 21288 tablet_service.cc:1467] Processing CreateTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e (DEFAULT_TABLE table=test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:04.909046 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935dc492de6e49b0ba969f7d5d5f330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:04.909165 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935dc492de6e49b0ba969f7d5d5f330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:04.909165 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935dc492de6e49b0ba969f7d5d5f330e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:04.911109 21458 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Bootstrap starting.
I20260924 20:20:04.912036 21459 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Bootstrap starting.
I20260924 20:20:04.912087 21458 tablet_bootstrap.cc:654] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:04.912972 21459 tablet_bootstrap.cc:654] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:04.913331 21458 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: No bootstrap required, opened a new log
I20260924 20:20:04.913435 21458 ts_tablet_manager.cc:1397] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:04.913727 21460 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Bootstrap starting.
I20260924 20:20:04.913895 21458 raft_consensus.cc:348] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.914023 21458 raft_consensus.cc:374] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:04.914067 21458 raft_consensus.cc:729] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154ab097046460e8e08a1a782230329, State: Initialized, Role: FOLLOWER
I20260924 20:20:04.914204 21458 consensus_queue.cc:260] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.914608 21458 ts_tablet_manager.cc:1428] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:04.914660 21460 tablet_bootstrap.cc:654] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:04.915426 21459 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: No bootstrap required, opened a new log
I20260924 20:20:04.915534 21459 ts_tablet_manager.cc:1397] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 20:20:04.916241 21459 raft_consensus.cc:348] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.916360 21459 raft_consensus.cc:374] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:04.916397 21459 raft_consensus.cc:729] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89873710cef64717ae120ba8dc111dfe, State: Initialized, Role: FOLLOWER
I20260924 20:20:04.916522 21459 consensus_queue.cc:260] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.916608 21460 tablet_bootstrap.cc:492] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: No bootstrap required, opened a new log
I20260924 20:20:04.916706 21460 ts_tablet_manager.cc:1397] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:04.916918 21459 ts_tablet_manager.cc:1428] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:04.917168 21460 raft_consensus.cc:348] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.917292 21460 raft_consensus.cc:374] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:04.917356 21460 raft_consensus.cc:729] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Initialized, Role: FOLLOWER
I20260924 20:20:04.917518 21460 consensus_queue.cc:260] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.917856 21460 ts_tablet_manager.cc:1428] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:20:04.940661 21442 raft_consensus.cc:482] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:04.940799 21442 raft_consensus.cc:504] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.941212 21442 leader_election.cc:288] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:04.941658 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935dc492de6e49b0ba969f7d5d5f330e" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe" is_pre_election: true
I20260924 20:20:04.941640 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935dc492de6e49b0ba969f7d5d5f330e" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329" is_pre_election: true
I20260924 20:20:04.941803 21028 raft_consensus.cc:2454] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:04.941809 21167 raft_consensus.cc:2454] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 0.
I20260924 20:20:04.942138 21241 leader_election.cc:302] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329; no voters:
I20260924 20:20:04.942335 21442 raft_consensus.cc:2781] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:04.942414 21442 raft_consensus.cc:482] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:04.942462 21442 raft_consensus.cc:3037] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:04.943702 21442 raft_consensus.cc:504] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.944047 21442 leader_election.cc:288] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Requested vote from peers 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459), 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033)
I20260924 20:20:04.944396 21028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935dc492de6e49b0ba969f7d5d5f330e" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4154ab097046460e8e08a1a782230329"
I20260924 20:20:04.944427 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935dc492de6e49b0ba969f7d5d5f330e" candidate_uuid: "32a86dea1a254f08b45c995d39384666" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe"
I20260924 20:20:04.944510 21028 raft_consensus.cc:3037] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:04.944561 21167 raft_consensus.cc:3037] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:04.945403 21028 raft_consensus.cc:2454] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:04.945657 21167 raft_consensus.cc:2454] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a86dea1a254f08b45c995d39384666 in term 1.
I20260924 20:20:04.945683 21241 leader_election.cc:302] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329; no voters:
I20260924 20:20:04.945847 21442 raft_consensus.cc:2781] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:04.945981 21442 raft_consensus.cc:686] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Becoming Leader. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Running, Role: LEADER
I20260924 20:20:04.946130 21442 consensus_queue.cc:237] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:04.947830 20871 catalog_manager.cc:5495] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a86dea1a254f08b45c995d39384666 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "32a86dea1a254f08b45c995d39384666" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } health_report { overall_health: HEALTHY } } }
I20260924 20:20:04.963795 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:04.960455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55362 (local address 127.18.172.67:37741)
0924 20:20:04.960789 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:04.960796 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:04.960806 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:20:04.960839 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:04.960841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:04.960879 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:04.960974 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:04.961235 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:04.961756 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:04.962625 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:04.962835 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:04.962936 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:04.963281 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:04.963350 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:04.963434 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:04.963524 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 20:20:04.963573 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":175,"threads_started":1}
I20260924 20:20:04.970322 21167 raft_consensus.cc:1261] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:04.970320 21028 raft_consensus.cc:1261] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Refusing update from remote peer 32a86dea1a254f08b45c995d39384666: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:04.970816 21442 consensus_queue.cc:1035] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:04.971026 21433 consensus_queue.cc:1035] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:05.002213 20870 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:05.002357 20870 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:05.004456 20870 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Sending DeleteTablet for 3 replicas of tablet 935dc492de6e49b0ba969f7d5d5f330e
I20260924 20:20:05.005079 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59638
I20260924 20:20:05.005156 21148 tablet_service.cc:1514] Processing DeleteTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 20:20:05.005290 21447 tablet_replica.cc:330] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: stopping tablet replica
I20260924 20:20:05.005342 21449 tablet_replica.cc:330] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: stopping tablet replica
I20260924 20:20:05.005450 21447 raft_consensus.cc:2229] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:05.005461 21449 raft_consensus.cc:2229] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:05.005592 21449 raft_consensus.cc:2258] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:05.005604 21447 raft_consensus.cc:2258] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:05.005713 21288 tablet_service.cc:1514] Processing DeleteTablet for tablet 935dc492de6e49b0ba969f7d5d5f330e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:04 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:55332
I20260924 20:20:05.005884 21448 tablet_replica.cc:330] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: stopping tablet replica
I20260924 20:20:05.006007 21448 raft_consensus.cc:2229] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:05.006224 21448 raft_consensus.cc:2258] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:05.006376 21449 ts_tablet_manager.cc:1905] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:05.006474 21447 ts_tablet_manager.cc:1905] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:05.006844 21448 ts_tablet_manager.cc:1905] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:05.006424 20870 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:05.007208 20870 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:05.009090 21447 ts_tablet_manager.cc:1918] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:05.009166 21447 log.cc:1196] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/935dc492de6e49b0ba969f7d5d5f330e
I20260924 20:20:05.009212 21449 ts_tablet_manager.cc:1918] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:05.009289 21449 log.cc:1196] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/935dc492de6e49b0ba969f7d5d5f330e
I20260924 20:20:05.009333 21448 ts_tablet_manager.cc:1918] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:05.009399 21448 log.cc:1196] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/935dc492de6e49b0ba969f7d5d5f330e
I20260924 20:20:05.009430 21447 ts_tablet_manager.cc:1939] T 935dc492de6e49b0ba969f7d5d5f330e P 4154ab097046460e8e08a1a782230329: Deleting consensus metadata
I20260924 20:20:05.009583 21449 ts_tablet_manager.cc:1939] T 935dc492de6e49b0ba969f7d5d5f330e P 89873710cef64717ae120ba8dc111dfe: Deleting consensus metadata
I20260924 20:20:05.009646 21448 ts_tablet_manager.cc:1939] T 935dc492de6e49b0ba969f7d5d5f330e P 32a86dea1a254f08b45c995d39384666: Deleting consensus metadata
I20260924 20:20:05.010205 20856 catalog_manager.cc:4841] TS 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741): tablet 935dc492de6e49b0ba969f7d5d5f330e (table test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]) successfully deleted
I20260924 20:20:05.010452 20858 catalog_manager.cc:4841] TS 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033): tablet 935dc492de6e49b0ba969f7d5d5f330e (table test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]) successfully deleted
I20260924 20:20:05.010872 20857 catalog_manager.cc:4841] TS 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459): tablet 935dc492de6e49b0ba969f7d5d5f330e (table test-table [id=f7d31d02c74444cdb22e56c01e00ce1c]) successfully deleted
I20260924 20:20:05.012194 21008 tablet_service.cc:1467] Processing CreateTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 (DEFAULT_TABLE table=test-table [id=274c7c73d288481c9e636c4682a0898e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:05.012322 21288 tablet_service.cc:1467] Processing CreateTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 (DEFAULT_TABLE table=test-table [id=274c7c73d288481c9e636c4682a0898e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:05.012465 21008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f60db4486b4fb1b9de63ea2164fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:05.012603 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f60db4486b4fb1b9de63ea2164fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:05.012602 21148 tablet_service.cc:1467] Processing CreateTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 (DEFAULT_TABLE table=test-table [id=274c7c73d288481c9e636c4682a0898e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:05.012845 21148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5f60db4486b4fb1b9de63ea2164fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:05.014551 21460 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Bootstrap starting.
I20260924 20:20:05.014878 21458 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: Bootstrap starting.
I20260924 20:20:05.015262 21459 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Bootstrap starting.
I20260924 20:20:05.015520 21460 tablet_bootstrap.cc:654] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:05.015951 21458 tablet_bootstrap.cc:654] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:05.016306 21459 tablet_bootstrap.cc:654] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:05.017355 21458 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: No bootstrap required, opened a new log
I20260924 20:20:05.017462 21458 ts_tablet_manager.cc:1397] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:05.017977 21460 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: No bootstrap required, opened a new log
I20260924 20:20:05.018097 21459 tablet_bootstrap.cc:492] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: No bootstrap required, opened a new log
I20260924 20:20:05.018106 21460 ts_tablet_manager.cc:1397] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 20:20:05.018065 21458 raft_consensus.cc:348] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.018175 21459 ts_tablet_manager.cc:1397] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:05.018184 21458 raft_consensus.cc:374] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:05.018220 21458 raft_consensus.cc:729] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4154ab097046460e8e08a1a782230329, State: Initialized, Role: FOLLOWER
I20260924 20:20:05.018352 21458 consensus_queue.cc:260] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.018548 21460 raft_consensus.cc:348] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.018659 21458 ts_tablet_manager.cc:1428] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:20:05.018666 21460 raft_consensus.cc:374] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:05.018702 21460 raft_consensus.cc:729] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a86dea1a254f08b45c995d39384666, State: Initialized, Role: FOLLOWER
I20260924 20:20:05.018838 21460 consensus_queue.cc:260] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.018918 21459 raft_consensus.cc:348] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.019063 21459 raft_consensus.cc:374] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:05.019111 21459 raft_consensus.cc:729] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89873710cef64717ae120ba8dc111dfe, State: Initialized, Role: FOLLOWER
I20260924 20:20:05.019250 21459 consensus_queue.cc:260] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.019555 21459 ts_tablet_manager.cc:1428] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:05.019649 21460 ts_tablet_manager.cc:1428] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:05.433691 21463 raft_consensus.cc:482] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:05.433852 21463 raft_consensus.cc:504] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.434994 21463 leader_election.cc:288] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033), 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741)
I20260924 20:20:05.439430 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.435418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:40059 (local address 127.18.172.67:37741)
0924 20:20:05.435559 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:05.435565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:05.436064 (+ 499us) server_negotiation.cc:407] Connection header received
0924 20:20:05.436128 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:05.436131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:05.436176 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:05.436267 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:05.436560 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.437105 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.438280 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.438791 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.438935 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.439030 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:05.439239 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 20:20:05.439312 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:20:05.439453 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.435310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37741 (local address 127.18.172.65:40059)
0924 20:20:05.435935 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:05.435949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:05.436047 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:05.436285 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:05.436287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:05.436302 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:05.436447 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.436453 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.437251 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.437254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.438152 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.438162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.438935 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.438939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.439064 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.439093 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:20:05.439182 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:20:05.439278 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":270,"thread_start_us":111,"threads_started":1}
I20260924 20:20:05.439677 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.435432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:55057 (local address 127.18.172.66:36033)
0924 20:20:05.435740 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:05.435753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:05.435771 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:20:05.435863 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:05.435866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:05.435934 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:05.436036 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:05.436374 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.437015 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.438405 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.438989 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.439164 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.439217 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:05.439438 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 20:20:05.439510 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
I20260924 20:20:05.439816 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.435310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36033 (local address 127.18.172.65:55057)
0924 20:20:05.435650 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:05.435662 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:05.435774 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:05.436091 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:05.436094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:05.436116 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:05.436255 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.436264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.437132 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.437138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.438270 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.438281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.439159 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.439162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.439314 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.439332 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:20:05.439422 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:20:05.439490 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":78,"thread_start_us":91,"threads_started":1}
I20260924 20:20:05.440064 21308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f60db4486b4fb1b9de63ea2164fbd3" candidate_uuid: "4154ab097046460e8e08a1a782230329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a86dea1a254f08b45c995d39384666" is_pre_election: true
I20260924 20:20:05.440207 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f60db4486b4fb1b9de63ea2164fbd3" candidate_uuid: "4154ab097046460e8e08a1a782230329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe" is_pre_election: true
I20260924 20:20:05.440322 21167 raft_consensus.cc:2454] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4154ab097046460e8e08a1a782230329 in term 0.
I20260924 20:20:05.440338 21308 raft_consensus.cc:2454] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4154ab097046460e8e08a1a782230329 in term 0.
I20260924 20:20:05.440809 20962 leader_election.cc:302] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4154ab097046460e8e08a1a782230329, 89873710cef64717ae120ba8dc111dfe; no voters:
I20260924 20:20:05.441143 21463 raft_consensus.cc:2781] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:05.441224 21463 raft_consensus.cc:482] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:05.441282 21463 raft_consensus.cc:3037] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:05.442685 21463 raft_consensus.cc:504] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.443147 21463 leader_election.cc:288] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [CANDIDATE]: Term 1 election: Requested vote from peers 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033), 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741)
I20260924 20:20:05.443652 21167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f60db4486b4fb1b9de63ea2164fbd3" candidate_uuid: "4154ab097046460e8e08a1a782230329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89873710cef64717ae120ba8dc111dfe"
I20260924 20:20:05.443677 21308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5f60db4486b4fb1b9de63ea2164fbd3" candidate_uuid: "4154ab097046460e8e08a1a782230329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a86dea1a254f08b45c995d39384666"
I20260924 20:20:05.443799 21167 raft_consensus.cc:3037] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:05.443804 21308 raft_consensus.cc:3037] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:05.444766 21308 raft_consensus.cc:2454] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4154ab097046460e8e08a1a782230329 in term 1.
I20260924 20:20:05.444768 21167 raft_consensus.cc:2454] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4154ab097046460e8e08a1a782230329 in term 1.
I20260924 20:20:05.445078 20960 leader_election.cc:302] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a86dea1a254f08b45c995d39384666, 4154ab097046460e8e08a1a782230329; no voters:
I20260924 20:20:05.445254 21463 raft_consensus.cc:2781] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:05.445508 21463 raft_consensus.cc:686] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [term 1 LEADER]: Becoming Leader. State: Replica: 4154ab097046460e8e08a1a782230329, State: Running, Role: LEADER
I20260924 20:20:05.445806 21463 consensus_queue.cc:237] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } }
I20260924 20:20:05.448388 20871 catalog_manager.cc:5495] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4154ab097046460e8e08a1a782230329 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "4154ab097046460e8e08a1a782230329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4154ab097046460e8e08a1a782230329" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:05.482208 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.478208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.18.172.65:33459)
0924 20:20:05.478473 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:05.478479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:05.478493 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:05.478582 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:05.478584 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:05.478639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:05.478743 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:05.478948 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.479660 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.480519 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.480835 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.480934 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.481292 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:05.481784 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:05.481877 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:05.481991 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 20:20:05.482041 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":122,"threads_started":1}
I20260924 20:20:05.492983 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.489706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33167 (local address 127.18.172.67:37741)
0924 20:20:05.489853 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:05.489856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:05.490004 (+ 148us) server_negotiation.cc:407] Connection header received
0924 20:20:05.490112 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:05.490114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:05.490182 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:05.490248 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:05.490466 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.490969 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.491903 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.492129 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.492231 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.492496 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:05.492569 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:05.492623 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:05.492795 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 20:20:05.492843 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:20:05.493063 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.489609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37741 (local address 127.18.172.65:33167)
0924 20:20:05.489923 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:05.489936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:05.490025 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:05.490271 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:05.490272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:05.490278 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:05.490378 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.490381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.491115 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.491118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.491744 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.491752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.492267 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.492270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.492338 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.492381 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:05.492685 (+ 304us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:05.492693 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:20:05.492758 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 20:20:05.492874 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":124,"threads_started":1}
I20260924 20:20:05.499424 21167 raft_consensus.cc:1261] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Refusing update from remote peer 4154ab097046460e8e08a1a782230329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:05.499974 21308 raft_consensus.cc:1261] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Refusing update from remote peer 4154ab097046460e8e08a1a782230329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:05.499954 21463 consensus_queue.cc:1035] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89873710cef64717ae120ba8dc111dfe" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:05.500481 21463 consensus_queue.cc:1035] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32a86dea1a254f08b45c995d39384666" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:05.518014 21482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.514148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:49153 (local address 127.18.172.65:33459)
0924 20:20:05.514295 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:05.514300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:05.514427 (+ 127us) server_negotiation.cc:407] Connection header received
0924 20:20:05.514619 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:05.514625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:05.514679 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:05.514772 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:05.515021 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.515688 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.516757 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.517013 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.517115 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.517522 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:05.517618 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:05.517696 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:05.517835 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 20:20:05.517898 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:05.518476 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:05.514081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33459 (local address 127.18.172.67:49153)
0924 20:20:05.514345 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:05.514362 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:05.514485 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:05.514788 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:05.514791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:05.514800 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:05.514915 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.514920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.515794 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.515797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.516609 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:05.516621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:05.517228 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:05.517230 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:05.517337 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:05.517367 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:05.517717 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:05.517722 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:20:05.518220 (+ 498us) client_negotiation.cc:240] Negotiation successful
0924 20:20:05.518319 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":73,"threads_started":1}
W20260924 20:20:05.519372 21429 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 20:20:07.085645 20869 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:07.085801 20869 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:07.088598 20869 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1f448be720941f9943cc12224a1c94c: Sending DeleteTablet for 3 replicas of tablet d5f60db4486b4fb1b9de63ea2164fbd3
I20260924 20:20:07.089228 21008 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59638
I20260924 20:20:07.089254 21148 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 20:20:07.089293 21288 tablet_service.cc:1514] Processing DeleteTablet for tablet d5f60db4486b4fb1b9de63ea2164fbd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:55332
I20260924 20:20:07.089625 21516 tablet_replica.cc:330] T d5f60db4486b4fb1b9de63ea2164fbd3 P 4154ab097046460e8e08a1a782230329: stopping tablet replica
I20260924 20:20:07.089653 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20943
I20260924 20:20:07.090893 21517 tablet_replica.cc:330] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: stopping tablet replica
I20260924 20:20:07.091039 21517 raft_consensus.cc:2229] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:07.091168 21517 raft_consensus.cc:2258] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:07.091420 21518 tablet_replica.cc:330] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: stopping tablet replica
I20260924 20:20:07.091557 21518 raft_consensus.cc:2229] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:07.091691 21518 raft_consensus.cc:2258] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:07.092324 21518 ts_tablet_manager.cc:1905] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:07.092324 21517 ts_tablet_manager.cc:1905] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:07.095054 21518 ts_tablet_manager.cc:1918] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:20:07.095054 21517 ts_tablet_manager.cc:1918] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 20:20:07.095135 21517 log.cc:1196] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/d5f60db4486b4fb1b9de63ea2164fbd3
I20260924 20:20:07.095202 21518 log.cc:1196] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/d5f60db4486b4fb1b9de63ea2164fbd3
I20260924 20:20:07.095432 21517 ts_tablet_manager.cc:1939] T d5f60db4486b4fb1b9de63ea2164fbd3 P 89873710cef64717ae120ba8dc111dfe: Deleting consensus metadata
I20260924 20:20:07.095486 21518 ts_tablet_manager.cc:1939] T d5f60db4486b4fb1b9de63ea2164fbd3 P 32a86dea1a254f08b45c995d39384666: Deleting consensus metadata
I20260924 20:20:07.096397 20856 catalog_manager.cc:4841] TS 32a86dea1a254f08b45c995d39384666 (127.18.172.67:37741): tablet d5f60db4486b4fb1b9de63ea2164fbd3 (table test-table [id=274c7c73d288481c9e636c4682a0898e]) successfully deleted
I20260924 20:20:07.096628 20858 catalog_manager.cc:4841] TS 89873710cef64717ae120ba8dc111dfe (127.18.172.66:36033): tablet d5f60db4486b4fb1b9de63ea2164fbd3 (table test-table [id=274c7c73d288481c9e636c4682a0898e]) successfully deleted
W20260924 20:20:07.097766 20857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:33459 (error 108)
I20260924 20:20:07.098011 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21083
I20260924 20:20:07.099303 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:07.098770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33459 (local address 127.0.0.1:41494)
0924 20:20:07.099077 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:07.099169 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:33459: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":84,"threads_started":1}
W20260924 20:20:07.101851 20857 catalog_manager.cc:4584] TS 4154ab097046460e8e08a1a782230329 (127.18.172.65:33459): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5f60db4486b4fb1b9de63ea2164fbd3: Network error: Client connection negotiation failed: client connection to 127.18.172.65:33459: connect: Connection refused (error 111)
I20260924 20:20:07.104178 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21223
I20260924 20:20:07.110934 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 20839
2026-09-24T20:20:07Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6421 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:07 dist-test-slave-m1qq krb5kdc[21523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:07 dist-test-slave-m1qq krb5kdc[21523](info): set up 2 sockets
Sep 24 20:20:07 dist-test-slave-m1qq krb5kdc[21523](info): commencing operation
krb5kdc: starting...
W20260924 20:20:08.160080 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:08.290149 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44607
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:08.401242 21536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:08.401515 21536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:08.401554 21536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:08.405643 21536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:08.405699 21536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:08.405718 21536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:08.405735 21536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:08.405750 21536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:08.411054 21536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:08.412132 21536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:08.413168 21536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:08.422212 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:08.422305 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.422276 21536 server_base.cc:1034] running on GCE node
W20260924 20:20:08.422287 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.423135 21536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:08.424378 21536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:08.425661 21536 hybrid_clock.cc:648] HybridClock initialized: now 1790281208425647 us; error 27 us; skew 500 ppm
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:08.429001 21536 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:08.429970 21536 webserver.cc:460] Webserver started at http://127.18.172.126:43971/ using document root <none> and password file <none>
I20260924 20:20:08.430555 21536 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:08.430658 21536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:08.430892 21536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:08.432988 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "cb33fd8db820400cac9910ef38ff9001"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "4f8af0bb9fe07e9a8f0cbf9de1044399"
server_key_iv: "7c4c81b00e50ade8d7d1b902b4fb150a"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.433560 21536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "cb33fd8db820400cac9910ef38ff9001"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "4f8af0bb9fe07e9a8f0cbf9de1044399"
server_key_iv: "7c4c81b00e50ade8d7d1b902b4fb150a"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.437359 21536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:20:08.439934 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.441226 21536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:20:08.441375 21536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "cb33fd8db820400cac9910ef38ff9001"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "4f8af0bb9fe07e9a8f0cbf9de1044399"
server_key_iv: "7c4c81b00e50ade8d7d1b902b4fb150a"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.441499 21536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:08.457242 21536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:08.460351 21536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:08.460554 21536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:08.467669 21536 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44607
I20260924 20:20:08.467684 21603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44607 every 8 connection(s)
I20260924 20:20:08.468618 21536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:08.471356 21604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:08.476796 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001: Bootstrap starting.
I20260924 20:20:08.477087 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21536
I20260924 20:20:08.477221 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:08.477485 19121 external_mini_cluster.cc:1428] Setting key 65a0da91b5ca54b0a52695b7cb2e69b3
I20260924 20:20:08.479537 21604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:08.480290 21604 log.cc:824] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:08.482201 21604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001: No bootstrap required, opened a new log
I20260924 20:20:08.486212 21604 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } }
I20260924 20:20:08.486536 21604 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:08.486598 21604 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb33fd8db820400cac9910ef38ff9001, State: Initialized, Role: FOLLOWER
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:08.487145 21604 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } }
I20260924 20:20:08.487315 21604 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:08.487393 21604 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:08.487514 21604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:08.488701 21604 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } }
I20260924 20:20:08.489110 21604 leader_election.cc:302] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb33fd8db820400cac9910ef38ff9001; no voters:
I20260924 20:20:08.489452 21604 leader_election.cc:288] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:08.489629 21609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:08.489864 21609 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [term 1 LEADER]: Becoming Leader. State: Replica: cb33fd8db820400cac9910ef38ff9001, State: Running, Role: LEADER
I20260924 20:20:08.490170 21609 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } }
I20260924 20:20:08.490873 21604 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:08.491451 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb33fd8db820400cac9910ef38ff9001. Latest consensus state: current_term: 1 leader_uuid: "cb33fd8db820400cac9910ef38ff9001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } } }
I20260924 20:20:08.491678 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:08.492235 21617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:08.494091 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:08.478855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.18.172.126:44607)
0924 20:20:08.479336 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:08.479351 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:08.479389 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:20:08.480207 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:08.480224 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:08.480621 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:08.480963 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:08.481802 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.482853 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.483752 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.484239 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.484412 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:08.487601 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:08.487618 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:08.487643 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:08.487688 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:08.489608 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:08.490399 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:08.490405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:08.490411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:08.490549 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:08.490872 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:08.490875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:08.490876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:08.491447 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:08.491628 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:08.492748 (+ 1120us) server_negotiation.cc:299] Negotiation successful
0924 20:20:08.493062 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":163,"threads_started":1}
I20260924 20:20:08.494074 21611 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb33fd8db820400cac9910ef38ff9001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb33fd8db820400cac9910ef38ff9001" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44607 } } }
I20260924 20:20:08.495401 21611 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:08.495550 21617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:08.501296 21617 catalog_manager.cc:1347] Generated new cluster ID: 456e96bcf2664a8dbfab632b9e32d212
I20260924 20:20:08.501359 21617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:08.523048 21617 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:08.524346 21617 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:08.532559 21617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb33fd8db820400cac9910ef38ff9001: Generated new TSK 0
I20260924 20:20:08.533301 21617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:08.570681 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:08.686062 21630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:08.686410 21630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:08.686487 21630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:08.690788 21630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:08.690876 21630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:08.690974 21630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:08.696676 21630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:08.697844 21630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:08.698925 21630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:08.708814 21635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.708899 21630 server_base.cc:1034] running on GCE node
W20260924 20:20:08.708863 21636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:08.708825 21638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.709489 21630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:08.710075 21630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:08.711238 21630 hybrid_clock.cc:648] HybridClock initialized: now 1790281208711223 us; error 26 us; skew 500 ppm
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:08.713932 21630 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:08.714795 21630 webserver.cc:460] Webserver started at http://127.18.172.65:42949/ using document root <none> and password file <none>
I20260924 20:20:08.715400 21630 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:08.715488 21630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:08.715709 21630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:08.717751 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "fd0c70c2ec5d4eac93ed0d50e1d060d3"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "39cba68f236d58920de1dca77a247dc2"
server_key_iv: "36fa06c09376748bd89e76e077031992"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.718302 21630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "fd0c70c2ec5d4eac93ed0d50e1d060d3"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "39cba68f236d58920de1dca77a247dc2"
server_key_iv: "36fa06c09376748bd89e76e077031992"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.722059 21630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:08.724519 21645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.725744 21630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:08.725872 21630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fd0c70c2ec5d4eac93ed0d50e1d060d3"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "39cba68f236d58920de1dca77a247dc2"
server_key_iv: "36fa06c09376748bd89e76e077031992"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.725980 21630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:08.738227 21630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:08.741283 21630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:08.741488 21630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:08.742063 21630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:08.742852 21630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:08.742923 21630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.742990 21630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:08.743072 21630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.752111 21630 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:45323
I20260924 20:20:08.752120 21758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:45323 every 8 connection(s)
I20260924 20:20:08.753002 21630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:08.757402 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21630
I20260924 20:20:08.757547 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:08.757855 19121 external_mini_cluster.cc:1428] Setting key 13e18ca5094772b827cbf68d500e57e8
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:08.768421 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:08.754823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:58317 (local address 127.18.172.126:44607)
0924 20:20:08.755022 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:08.755028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:08.755830 (+ 802us) server_negotiation.cc:407] Connection header received
0924 20:20:08.757105 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:08.757109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:08.757158 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:08.757295 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:08.758876 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.759634 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.760903 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.761185 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.761311 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:08.764328 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:08.764332 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:08.764335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:08.764371 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:08.765989 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:08.766558 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:08.766562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:08.766564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:08.766653 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:08.767112 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:08.767117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:08.767119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:08.767329 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:08.767448 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:08.768077 (+ 629us) server_negotiation.cc:299] Negotiation successful
0924 20:20:08.768225 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:08.769594 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:08.755243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44607 (local address 127.18.172.65:58317)
0924 20:20:08.755671 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:08.755715 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:08.756874 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:08.757511 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:08.757521 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:08.758026 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:08.758724 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:08.758745 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.759764 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.759767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:08.760760 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:08.760770 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:08.761350 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:08.761355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:08.761554 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:08.762235 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:08.762254 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:08.764121 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:08.766123 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:08.766130 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:08.766148 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:08.766436 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:08.766757 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:08.766762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:08.766764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:08.766983 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:08.767446 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:08.767454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:08.767741 (+ 287us) client_negotiation.cc:769] Sending connection context
0924 20:20:08.768007 (+ 266us) client_negotiation.cc:240] Negotiation successful
0924 20:20:08.768381 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":111,"threads_started":1}
I20260924 20:20:08.770941 21759 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44607
I20260924 20:20:08.771215 21759 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:08.771719 21759 heartbeater.cc:511] Master 127.18.172.126:44607 requested a full tablet report, sending...
I20260924 20:20:08.773243 21568 ts_manager.cc:194] Registered new tserver with Master: fd0c70c2ec5d4eac93ed0d50e1d060d3 (127.18.172.65:45323)
I20260924 20:20:08.774412 21568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:58317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:08.786947 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:08.899688 21764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:08.899971 21764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:08.900051 21764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:08.904145 21764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:08.904227 21764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:08.904312 21764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:08.909796 21764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:08.910883 21764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:08.911935 21764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:08.921777 21772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:08.921777 21770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.921983 21764 server_base.cc:1034] running on GCE node
W20260924 20:20:08.922178 21769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:08.922585 21764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:08.923286 21764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:08.924438 21764 hybrid_clock.cc:648] HybridClock initialized: now 1790281208924416 us; error 36 us; skew 500 ppm
Sep 24 20:20:08 dist-test-slave-m1qq krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:08.927191 21764 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:08.928050 21764 webserver.cc:460] Webserver started at http://127.18.172.66:35233/ using document root <none> and password file <none>
I20260924 20:20:08.928596 21764 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:08.928663 21764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:08.928874 21764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:08.930872 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "b66726734b714f9a96ea026b57d7dbbd"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "0a48fc3547ea04edd896c81a7c7055c6"
server_key_iv: "c48549702dd5f8af3214834bcaa1edce"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.931463 21764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "b66726734b714f9a96ea026b57d7dbbd"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "0a48fc3547ea04edd896c81a7c7055c6"
server_key_iv: "c48549702dd5f8af3214834bcaa1edce"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.935119 21764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:08.937513 21779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.938757 21764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:08.938892 21764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "b66726734b714f9a96ea026b57d7dbbd"
format_stamp: "Formatted at 2026-09-24 20:20:08 on dist-test-slave-m1qq"
server_key: "0a48fc3547ea04edd896c81a7c7055c6"
server_key_iv: "c48549702dd5f8af3214834bcaa1edce"
server_key_version: "encryptionkey@0"
I20260924 20:20:08.939005 21764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:08.981855 21764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:08.984932 21764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:08.985141 21764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:08.985737 21764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:08.986573 21764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:08.986653 21764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.986728 21764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:08.986783 21764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:08.996097 21764 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:44051
I20260924 20:20:08.996124 21892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:44051 every 8 connection(s)
I20260924 20:20:08.997030 21764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:09.003558 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21764
I20260924 20:20:09.003684 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:09.003957 19121 external_mini_cluster.cc:1428] Setting key 2062d61f6dc02ec7f2bce230565a7fec
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[21523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281208, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:09.011333 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:08.998883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:49111 (local address 127.18.172.126:44607)
0924 20:20:08.999084 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:08.999087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:08.999928 (+ 841us) server_negotiation.cc:407] Connection header received
0924 20:20:09.001233 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:09.001237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:09.001293 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:09.001402 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:09.002646 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.003264 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.004621 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.004839 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.004921 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:09.007780 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:09.007783 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:09.007786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:09.007814 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:09.009293 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:09.009820 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:09.009824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:09.009826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:09.009906 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:09.010259 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:09.010264 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:09.010265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:09.010447 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:09.010560 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:09.011057 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 20:20:09.011174 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:09.012156 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:08.999292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44607 (local address 127.18.172.66:49111)
0924 20:20:08.999782 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:08.999826 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:09.001028 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:09.001671 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:09.001678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:09.002027 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:09.002506 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:09.002519 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.003417 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.003420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:09.004481 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:09.004493 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.004992 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.004996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:09.005183 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:09.005796 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:09.005820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:09.007608 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:09.009431 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:09.009437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:09.009453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:09.009715 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:09.010020 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:09.010023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:09.010025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:09.010166 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:09.010560 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:09.010566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:09.010808 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 20:20:09.010994 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 20:20:09.011332 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:09.013356 21893 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44607
I20260924 20:20:09.013609 21893 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:09.014050 21893 heartbeater.cc:511] Master 127.18.172.126:44607 requested a full tablet report, sending...
I20260924 20:20:09.015152 21568 ts_manager.cc:194] Registered new tserver with Master: b66726734b714f9a96ea026b57d7dbbd (127.18.172.66:44051)
I20260924 20:20:09.015717 21568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:49111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:09.030777 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:09.140745 21898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:09.140986 21898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:09.141055 21898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:09.145279 21898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:09.145339 21898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:09.145437 21898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:09.150722 21898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:09.151794 21898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:09.152818 21898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:09.163180 21904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:09.163220 21898 server_base.cc:1034] running on GCE node
W20260924 20:20:09.163221 21906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:09.163174 21903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:09.163815 21898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:09.164471 21898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:09.165666 21898 hybrid_clock.cc:648] HybridClock initialized: now 1790281209165636 us; error 57 us; skew 500 ppm
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[21523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281209, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:09.168540 21898 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:09.169489 21898 webserver.cc:460] Webserver started at http://127.18.172.67:34083/ using document root <none> and password file <none>
I20260924 20:20:09.170048 21898 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:09.170111 21898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:09.170274 21898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:09.172300 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "c9bea0522661470fb60c85a30fd370db"
format_stamp: "Formatted at 2026-09-24 20:20:09 on dist-test-slave-m1qq"
server_key: "c7c86a16e9caceba4ded6a3111b3f7f1"
server_key_iv: "fb3ec393fe789f044a6a30bf80af01ef"
server_key_version: "encryptionkey@0"
I20260924 20:20:09.172947 21898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "c9bea0522661470fb60c85a30fd370db"
format_stamp: "Formatted at 2026-09-24 20:20:09 on dist-test-slave-m1qq"
server_key: "c7c86a16e9caceba4ded6a3111b3f7f1"
server_key_iv: "fb3ec393fe789f044a6a30bf80af01ef"
server_key_version: "encryptionkey@0"
I20260924 20:20:09.176847 21898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:20:09.179302 21913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:09.180824 21898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:09.180948 21898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "c9bea0522661470fb60c85a30fd370db"
format_stamp: "Formatted at 2026-09-24 20:20:09 on dist-test-slave-m1qq"
server_key: "c7c86a16e9caceba4ded6a3111b3f7f1"
server_key_iv: "fb3ec393fe789f044a6a30bf80af01ef"
server_key_version: "encryptionkey@0"
I20260924 20:20:09.181062 21898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:09.194918 21898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:09.197948 21898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:09.198154 21898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:09.198752 21898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:09.199637 21898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:09.199681 21898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:09.199751 21898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:09.199782 21898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:09.208439 21898 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:44575
I20260924 20:20:09.208460 22026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:44575 every 8 connection(s)
I20260924 20:20:09.209328 21898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:09.217460 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 21898
I20260924 20:20:09.217590 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:09.217921 19121 external_mini_cluster.cc:1428] Setting key ede2403cc3e0e49067c7401b3b99dddb
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[21523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281209, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:09.222923 21607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:09.211137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:60255 (local address 127.18.172.126:44607)
0924 20:20:09.211288 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:09.211292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:09.212054 (+ 762us) server_negotiation.cc:407] Connection header received
0924 20:20:09.213370 (+ 1316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:09.213374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:09.213431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:09.213509 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:09.214652 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.215252 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.216366 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.216606 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.216697 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:09.219613 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:09.219615 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:09.219618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:09.219641 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:09.221178 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:09.221644 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:09.221647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:09.221648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:09.221720 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:09.222003 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:09.222005 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:09.222006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:09.222182 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:09.222254 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:09.222716 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 20:20:09.222806 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:09.223887 22029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:09.211475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44607 (local address 127.18.172.67:60255)
0924 20:20:09.211909 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:09.211950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:09.213162 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:09.213679 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:09.213685 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:09.214022 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:09.214479 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:09.214491 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.215365 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.215368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:09.216248 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:09.216257 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:09.216744 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:09.216746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:09.216879 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:09.217435 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:09.217582 (+ 147us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:09.219422 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:09.221279 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:09.221283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:09.221296 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:09.221555 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:09.221797 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:09.221799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:09.221800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:09.221926 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:09.222266 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:09.222270 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:09.222497 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 20:20:09.222671 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 20:20:09.222932 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":123,"threads_started":1}
I20260924 20:20:09.225008 22027 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44607
I20260924 20:20:09.225229 22027 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:09.225669 22027 heartbeater.cc:511] Master 127.18.172.126:44607 requested a full tablet report, sending...
I20260924 20:20:09.226562 21568 ts_manager.cc:194] Registered new tserver with Master: c9bea0522661470fb60c85a30fd370db (127.18.172.67:44575)
I20260924 20:20:09.227196 21568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:60255
I20260924 20:20:09.232766 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:20:09.246397 21607 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:09.244942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.18.172.126:44607)
0924 20:20:09.245074 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:09.245078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:09.245230 (+ 152us) server_negotiation.cc:407] Connection header received
0924 20:20:09.245407 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:09.245409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:09.245450 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:09.245524 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:09.246283 (+ 759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49798: BlockingRecv error: recv got EOF from 127.0.0.1:49798 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:09.247586 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21630
I20260924 20:20:09.253417 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21764
I20260924 20:20:09.258759 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21898
I20260924 20:20:09.263870 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 21536
2026-09-24T20:20:09Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2151 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[22039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[22039](info): set up 2 sockets
Sep 24 20:20:09 dist-test-slave-m1qq krb5kdc[22039](info): commencing operation
krb5kdc: starting...
W20260924 20:20:10.344170 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:10.470923 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35623
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:10.584515 22052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:10.584795 22052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:10.584837 22052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:10.589629 22052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:10.589716 22052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:10.589749 22052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:10.589784 22052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:10.589818 22052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:10.595907 22052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:35623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:10.597018 22052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:10.598078 22052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:10.606523 22057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:10.606523 22058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:10.606523 22060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:10.606740 22052 server_base.cc:1034] running on GCE node
I20260924 20:20:10.607543 22052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:10.608666 22052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:10.609849 22052 hybrid_clock.cc:648] HybridClock initialized: now 1790281210609820 us; error 43 us; skew 500 ppm
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281210, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:10.616353 22052 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:10.617264 22052 webserver.cc:460] Webserver started at http://127.18.172.126:40981/ using document root <none> and password file <none>
I20260924 20:20:10.617841 22052 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:10.617931 22052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:10.618165 22052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:10.620239 22052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "767a8cb3b36e46e7884d066462f142fd"
format_stamp: "Formatted at 2026-09-24 20:20:10 on dist-test-slave-m1qq"
server_key: "b3beefd6fea0a75e92409b5465c366a4"
server_key_iv: "e32c14025e995182dab120f6f98a5fb7"
server_key_version: "encryptionkey@0"
I20260924 20:20:10.620797 22052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "767a8cb3b36e46e7884d066462f142fd"
format_stamp: "Formatted at 2026-09-24 20:20:10 on dist-test-slave-m1qq"
server_key: "b3beefd6fea0a75e92409b5465c366a4"
server_key_iv: "e32c14025e995182dab120f6f98a5fb7"
server_key_version: "encryptionkey@0"
I20260924 20:20:10.624478 22052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:10.627065 22067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:10.628407 22052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:10.628557 22052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "767a8cb3b36e46e7884d066462f142fd"
format_stamp: "Formatted at 2026-09-24 20:20:10 on dist-test-slave-m1qq"
server_key: "b3beefd6fea0a75e92409b5465c366a4"
server_key_iv: "e32c14025e995182dab120f6f98a5fb7"
server_key_version: "encryptionkey@0"
I20260924 20:20:10.628685 22052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:10.650610 22052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:10.654009 22052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:10.654232 22052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:10.661363 22052 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:35623
I20260924 20:20:10.661659 22119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:35623 every 8 connection(s)
I20260924 20:20:10.662406 22052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:10.665114 22120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:10.667728 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22052
I20260924 20:20:10.667851 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:10.668129 19121 external_mini_cluster.cc:1428] Setting key 9994c5fcd48a8d74b86ab17e4fe94c8e
I20260924 20:20:10.671437 22120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd: Bootstrap starting.
I20260924 20:20:10.673601 22120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:10.674335 22120 log.cc:824] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:10.676214 22120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd: No bootstrap required, opened a new log
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:10.680387 22120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } }
I20260924 20:20:10.680615 22120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:10.680675 22120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767a8cb3b36e46e7884d066462f142fd, State: Initialized, Role: FOLLOWER
I20260924 20:20:10.681214 22120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } }
I20260924 20:20:10.681360 22120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:10.681443 22120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:10.681543 22120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:10.682973 22120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } }
I20260924 20:20:10.683425 22120 leader_election.cc:302] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767a8cb3b36e46e7884d066462f142fd; no voters:
I20260924 20:20:10.683784 22120 leader_election.cc:288] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:10.683892 22125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:10.684105 22125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [term 1 LEADER]: Becoming Leader. State: Replica: 767a8cb3b36e46e7884d066462f142fd, State: Running, Role: LEADER
I20260924 20:20:10.684342 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:10.669609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.18.172.126:35623)
0924 20:20:10.670086 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:10.670118 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:10.670153 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:10.670780 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:10.670797 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:10.671222 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:10.671565 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:10.672444 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:10.673415 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:10.674350 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:10.674878 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:10.675080 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:10.677855 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:10.677860 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:10.677873 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:10.677900 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:10.680724 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:10.681247 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:10.681252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:10.681257 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:10.681480 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:10.682021 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:10.682024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:10.682025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:10.682465 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:10.682651 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:10.682986 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 20:20:10.683352 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260924 20:20:10.684432 22125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } }
I20260924 20:20:10.685209 22120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:10.686189 22126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767a8cb3b36e46e7884d066462f142fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } } }
I20260924 20:20:10.686318 22126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:10.685554 22125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767a8cb3b36e46e7884d066462f142fd. Latest consensus state: current_term: 1 leader_uuid: "767a8cb3b36e46e7884d066462f142fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767a8cb3b36e46e7884d066462f142fd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35623 } } }
I20260924 20:20:10.686856 22125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:10.687878 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:20:10.687907 22134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:10.691140 22134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:10.698146 22134 catalog_manager.cc:1347] Generated new cluster ID: b8a343c6ad2541fdb8560fbff26ac345
I20260924 20:20:10.698226 22134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:10.706688 22123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:10.701647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33644 (local address 127.18.172.126:35623)
0924 20:20:10.701779 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:10.701784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:10.702058 (+ 274us) server_negotiation.cc:407] Connection header received
0924 20:20:10.702251 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:10.702254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:10.702296 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:10.702381 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:10.702962 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:10.703577 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:10.704450 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:10.704699 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:10.704812 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:10.705724 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:10.705727 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:10.705730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:10.706219 (+ 489us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:10.706249 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:10.706265 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:10.706369 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:10.706449 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 20:20:10.706518 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:10.715250 22134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:10.716352 22134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:10.729650 22134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767a8cb3b36e46e7884d066462f142fd: Generated new TSK 0
I20260924 20:20:10.730460 22134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:10.742231 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22052
2026-09-24T20:20:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1476 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22153](info): set up 2 sockets
Sep 24 20:20:10 dist-test-slave-m1qq krb5kdc[22153](info): commencing operation
krb5kdc: starting...
W20260924 20:20:11.780726 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:11 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:11.906329 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39643
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:12.018723 22166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:12.019109 22166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:12.019176 22166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:12.023478 22166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:12.023540 22166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:12.023561 22166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:12.023578 22166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:12.023594 22166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:12.029132 22166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:12.030254 22166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:12.031392 22166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:12.040637 22174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.040819 22166 server_base.cc:1034] running on GCE node
W20260924 20:20:12.040637 22171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.040637 22172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.041635 22166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:12.042798 22166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:12.043974 22166 hybrid_clock.cc:648] HybridClock initialized: now 1790281212043973 us; error 63 us; skew 500 ppm
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:12.047149 22166 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:12.048236 22166 webserver.cc:460] Webserver started at http://127.18.172.126:41937/ using document root <none> and password file <none>
I20260924 20:20:12.048836 22166 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:12.048898 22166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:12.049116 22166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:12.051291 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "3cca38f1ff144eff9048c26e3088fdac"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "1b65b0eca08ce6ea58aa400d35af0ce9"
server_key_iv: "2beb6e460a6356282a3259da90ba52dd"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.051877 22166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "3cca38f1ff144eff9048c26e3088fdac"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "1b65b0eca08ce6ea58aa400d35af0ce9"
server_key_iv: "2beb6e460a6356282a3259da90ba52dd"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.055788 22166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:20:12.058415 22181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.059775 22166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:12.059932 22166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "3cca38f1ff144eff9048c26e3088fdac"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "1b65b0eca08ce6ea58aa400d35af0ce9"
server_key_iv: "2beb6e460a6356282a3259da90ba52dd"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.060051 22166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:12.070371 22166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:12.074293 22166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:12.074517 22166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:12.081770 22166 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:39643
I20260924 20:20:12.082098 22233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:39643 every 8 connection(s)
I20260924 20:20:12.082772 22166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:12.085602 22234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:12.091253 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac: Bootstrap starting.
I20260924 20:20:12.092775 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22166
I20260924 20:20:12.092902 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:12.093206 19121 external_mini_cluster.cc:1428] Setting key 314f9ac68aa6ccc072806a271f8526c3
I20260924 20:20:12.094012 22234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:12.094767 22234 log.cc:824] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:12.096961 22234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac: No bootstrap required, opened a new log
I20260924 20:20:12.101259 22234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } }
I20260924 20:20:12.101583 22234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:12.101652 22234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cca38f1ff144eff9048c26e3088fdac, State: Initialized, Role: FOLLOWER
I20260924 20:20:12.102187 22234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } }
I20260924 20:20:12.102345 22234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:12.102417 22234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:12.102558 22234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 0 FOLLOWER]: Advancing to term 1
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:12.103793 22234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } }
I20260924 20:20:12.104167 22234 leader_election.cc:302] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cca38f1ff144eff9048c26e3088fdac; no voters:
I20260924 20:20:12.104497 22234 leader_election.cc:288] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:12.104632 22239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:12.104869 22239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [term 1 LEADER]: Becoming Leader. State: Replica: 3cca38f1ff144eff9048c26e3088fdac, State: Running, Role: LEADER
I20260924 20:20:12.105192 22239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } }
I20260924 20:20:12.105967 22234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:12.107048 22240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cca38f1ff144eff9048c26e3088fdac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } } }
I20260924 20:20:12.107196 22240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:12.107581 22248 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:12.109231 22241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cca38f1ff144eff9048c26e3088fdac. Latest consensus state: current_term: 1 leader_uuid: "3cca38f1ff144eff9048c26e3088fdac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cca38f1ff144eff9048c26e3088fdac" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39643 } } }
I20260924 20:20:12.109336 22241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:12.110189 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.094390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59100 (local address 127.18.172.126:39643)
0924 20:20:12.094920 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:12.094947 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:12.094982 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:12.095774 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:12.095793 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:12.096114 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:12.096393 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:12.097305 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.098346 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.099656 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.100172 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.100295 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.103417 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:12.103425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:12.103443 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:12.103477 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:12.105569 (+ 2092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.106139 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.106145 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.106151 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.106310 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.107688 (+ 1378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.107691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.107693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.108210 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:12.108416 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:12.108863 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 20:20:12.109201 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":156,"threads_started":1}
I20260924 20:20:12.110991 22248 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:12.117026 22248 catalog_manager.cc:1347] Generated new cluster ID: 8c05a5ca55774a1a80658ac1cb387757
I20260924 20:20:12.117101 22248 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:12.143585 22248 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:12.144611 22248 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:12.153581 22248 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cca38f1ff144eff9048c26e3088fdac: Generated new TSK 0
I20260924 20:20:12.154354 22248 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:12.190236 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:12.303802 22260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:12.304136 22260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:12.304224 22260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:12.308588 22260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:12.308676 22260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:12.308769 22260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:12.314145 22260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:12.315340 22260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:12.316432 22260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:12.326310 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.326363 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.326310 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.326671 22260 server_base.cc:1034] running on GCE node
I20260924 20:20:12.327116 22260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:12.327785 22260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:12.329000 22260 hybrid_clock.cc:648] HybridClock initialized: now 1790281212328983 us; error 27 us; skew 500 ppm
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:12.331699 22260 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:12.332510 22260 webserver.cc:460] Webserver started at http://127.18.172.65:45701/ using document root <none> and password file <none>
I20260924 20:20:12.333025 22260 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:12.333081 22260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:12.333262 22260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:12.335301 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "5804574203ed42bdb284c0f4346d0211"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "d19240ca262c45cae877ca997f18c4bb"
server_key_iv: "7d35e8d13ab19c18fbc28bfff50a814b"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.335820 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "5804574203ed42bdb284c0f4346d0211"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "d19240ca262c45cae877ca997f18c4bb"
server_key_iv: "7d35e8d13ab19c18fbc28bfff50a814b"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.339783 22260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:20:12.342176 22275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.343462 22260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:12.343643 22260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "5804574203ed42bdb284c0f4346d0211"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "d19240ca262c45cae877ca997f18c4bb"
server_key_iv: "7d35e8d13ab19c18fbc28bfff50a814b"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.343742 22260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:12.364776 22260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:12.367853 22260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:12.368067 22260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:12.368690 22260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:12.369534 22260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:12.369586 22260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.369658 22260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:12.369689 22260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.378643 22260 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:34693
I20260924 20:20:12.378711 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:34693 every 8 connection(s)
I20260924 20:20:12.379611 22260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:12.386965 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22260
I20260924 20:20:12.387188 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:12.387460 19121 external_mini_cluster.cc:1428] Setting key fbb86ae00c066fe0c25de0b35532ee91
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:12.394835 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.381398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:57171 (local address 127.18.172.126:39643)
0924 20:20:12.381552 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:12.381557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:12.382583 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 20:20:12.383987 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:12.383990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:12.384037 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:12.384120 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:12.385412 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.385939 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.387202 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.387492 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.387616 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.390757 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:12.390759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:12.390761 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:12.390784 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:12.392568 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.393072 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.393079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.393084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.393182 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.393507 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.393513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.393518 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.393797 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:12.393910 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:12.394420 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 20:20:12.394569 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:20:12.395449 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.381893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39643 (local address 127.18.172.65:57171)
0924 20:20:12.382393 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:12.382446 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:12.383766 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:12.384336 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:12.384344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:12.384745 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:12.385256 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.385271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.386056 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.386059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.386986 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.386995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.387665 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.387670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.387880 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.388540 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:12.388556 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:12.390552 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:12.392689 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.392694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.392707 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.392964 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.393277 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.393279 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.393281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.393409 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.393918 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:12.393923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:12.394169 (+ 246us) client_negotiation.cc:769] Sending connection context
0924 20:20:12.394329 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:20:12.394592 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":112,"threads_started":1}
I20260924 20:20:12.396934 22389 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39643
I20260924 20:20:12.397148 22389 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:12.397610 22389 heartbeater.cc:511] Master 127.18.172.126:39643 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:12.399199 22198 ts_manager.cc:194] Registered new tserver with Master: 5804574203ed42bdb284c0f4346d0211 (127.18.172.65:34693)
I20260924 20:20:12.400367 22198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:57171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:12.417522 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:12.528545 22394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:12.528851 22394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:12.528939 22394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:12.533092 22394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:12.533182 22394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:12.533269 22394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:12.538686 22394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:12.539863 22394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:12.540889 22394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:12.550877 22402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.550944 22394 server_base.cc:1034] running on GCE node
W20260924 20:20:12.550855 22399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.550863 22400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.551565 22394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:12.552153 22394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:12.553325 22394 hybrid_clock.cc:648] HybridClock initialized: now 1790281212553287 us; error 42 us; skew 500 ppm
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:12.556326 22394 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:12.557191 22394 webserver.cc:460] Webserver started at http://127.18.172.66:43759/ using document root <none> and password file <none>
I20260924 20:20:12.557736 22394 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:12.557796 22394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:12.557956 22394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:12.560063 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "00a792c52206485f8d44f528642aa205"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "dea42b46eabcf97676f2a8734434a9d7"
server_key_iv: "5f7b3156c76e0267f7013170edda2136"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.560580 22394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "00a792c52206485f8d44f528642aa205"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "dea42b46eabcf97676f2a8734434a9d7"
server_key_iv: "5f7b3156c76e0267f7013170edda2136"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.564631 22394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:20:12.567121 22409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.568449 22394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:12.568568 22394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "00a792c52206485f8d44f528642aa205"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "dea42b46eabcf97676f2a8734434a9d7"
server_key_iv: "5f7b3156c76e0267f7013170edda2136"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.568660 22394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:12.592232 22394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:12.595302 22394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:12.595454 22394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:12.596079 22394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:12.597246 22394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:12.597332 22394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.597388 22394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:12.597435 22394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.606930 22394 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:38239
I20260924 20:20:12.606972 22522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:38239 every 8 connection(s)
I20260924 20:20:12.607941 22394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:12.613976 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22394
I20260924 20:20:12.614104 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:12.614357 19121 external_mini_cluster.cc:1428] Setting key f48e016cc096d35c5cd882596e1e83fd
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:12.622828 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.609948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:51605 (local address 127.18.172.126:39643)
0924 20:20:12.610106 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:12.610109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:12.610902 (+ 793us) server_negotiation.cc:407] Connection header received
0924 20:20:12.612437 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:12.612440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:12.612498 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:12.612607 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:12.613761 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.614441 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.615761 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.615983 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.616088 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.618950 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:12.618953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:12.618955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:12.618978 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:12.620480 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.621137 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.621141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.621143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.621214 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.621674 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.621676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.621677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.621857 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:12.621933 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:12.622595 (+ 662us) server_negotiation.cc:299] Negotiation successful
0924 20:20:12.622707 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:20:12.624145 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.610319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39643 (local address 127.18.172.66:51605)
0924 20:20:12.610756 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:12.610798 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:12.612224 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:12.612779 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:12.612786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:12.613147 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:12.613617 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.613630 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.614567 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.614575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.615633 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.615643 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.616125 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.616129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.616301 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.616905 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:12.616921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:12.618780 (+ 1859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:12.620615 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.620624 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.620647 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.621037 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.621328 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.621332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.621337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.621537 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.621964 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:12.621971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:12.622326 (+ 355us) client_negotiation.cc:769] Sending connection context
0924 20:20:12.622570 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 20:20:12.622935 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:12.625340 22523 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39643
I20260924 20:20:12.625619 22523 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:12.626120 22523 heartbeater.cc:511] Master 127.18.172.126:39643 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:20:12.627240 22198 ts_manager.cc:194] Registered new tserver with Master: 00a792c52206485f8d44f528642aa205 (127.18.172.66:38239)
I20260924 20:20:12.627861 22198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:51605
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:12.642586 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:38277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:12.752985 22528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:12.753294 22528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:12.753340 22528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:12.757341 22528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:12.757459 22528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:12.757599 22528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:12.762780 22528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:38277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:12.763926 22528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:12.764981 22528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:12.774744 22534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.774931 22533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:12.774940 22536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:12.775211 22528 server_base.cc:1034] running on GCE node
I20260924 20:20:12.775617 22528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:12.776305 22528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:12.777487 22528 hybrid_clock.cc:648] HybridClock initialized: now 1790281212777434 us; error 66 us; skew 500 ppm
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:12.780225 22528 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:12.781113 22528 webserver.cc:460] Webserver started at http://127.18.172.67:39803/ using document root <none> and password file <none>
I20260924 20:20:12.781699 22528 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:12.781791 22528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:12.782018 22528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:12.784130 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "9cec5c50f64a4b0ebc7a5e615cb2701b"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "bd9ab27f788b70ef97e66cedf902f12e"
server_key_iv: "f748f4a2edc65605603adb6f8aa2cab0"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.784706 22528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "9cec5c50f64a4b0ebc7a5e615cb2701b"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "bd9ab27f788b70ef97e66cedf902f12e"
server_key_iv: "f748f4a2edc65605603adb6f8aa2cab0"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.788524 22528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:20:12.790947 22543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.792222 22528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:12.792354 22528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "9cec5c50f64a4b0ebc7a5e615cb2701b"
format_stamp: "Formatted at 2026-09-24 20:20:12 on dist-test-slave-m1qq"
server_key: "bd9ab27f788b70ef97e66cedf902f12e"
server_key_iv: "f748f4a2edc65605603adb6f8aa2cab0"
server_key_version: "encryptionkey@0"
I20260924 20:20:12.792470 22528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:12.818897 22528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:12.821987 22528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:12.822198 22528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:12.822813 22528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:12.823691 22528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:12.823761 22528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.823828 22528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:12.823874 22528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:12.832834 22528 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:43777
I20260924 20:20:12.832859 22656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:43777 every 8 connection(s)
I20260924 20:20:12.833726 22528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:12.838788 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22528
I20260924 20:20:12.838915 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:12.839228 19121 external_mini_cluster.cc:1428] Setting key 97b0985552a15ac5bdcc46c7d328db04
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:12.847767 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.835554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:35599 (local address 127.18.172.126:39643)
0924 20:20:12.835706 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:12.835709 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:12.836536 (+ 827us) server_negotiation.cc:407] Connection header received
0924 20:20:12.837925 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:12.837928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:12.837984 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:12.838068 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:12.839598 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.840262 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.841707 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.841946 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.842056 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.844451 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:12.844454 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:12.844456 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:12.844480 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:12.845899 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.846368 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.846372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.846373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.846455 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.846764 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.846767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.846768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.846930 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:12.847043 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:12.847549 (+ 506us) server_negotiation.cc:299] Negotiation successful
0924 20:20:12.847651 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:12.848623 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.835940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39643 (local address 127.18.172.67:35599)
0924 20:20:12.836387 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:12.836430 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:12.837708 (+ 1278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:12.838253 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:12.838261 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:12.838697 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:12.839431 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.839453 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.840389 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.840392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.841557 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:12.841565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.842092 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.842096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:12.842273 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.842790 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:12.842805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:12.844304 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:12.845999 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.846003 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.846014 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.846257 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.846538 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:12.846540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:12.846542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:12.846684 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:12.847062 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:12.847069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:12.847322 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 20:20:12.847501 (+ 179us) client_negotiation.cc:240] Negotiation successful
0924 20:20:12.847761 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260924 20:20:12.849876 22657 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39643
I20260924 20:20:12.850080 22657 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:12.850512 22657 heartbeater.cc:511] Master 127.18.172.126:39643 requested a full tablet report, sending...
I20260924 20:20:12.851545 22198 ts_manager.cc:194] Registered new tserver with Master: 9cec5c50f64a4b0ebc7a5e615cb2701b (127.18.172.67:43777)
I20260924 20:20:12.852326 22198 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:35599
I20260924 20:20:12.853240 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281212, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:12.880239 22237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:12.870134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.18.172.126:39643)
0924 20:20:12.870276 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:12.870279 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:12.870470 (+ 191us) server_negotiation.cc:407] Connection header received
0924 20:20:12.870591 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:12.870593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:12.870637 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:12.870715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:12.871466 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.872084 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.872988 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:12.873266 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:12.873347 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:12.876179 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:12.876184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:12.876186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:12.876214 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:12.878141 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.878632 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.878642 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.878644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.878734 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:12.879123 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:12.879128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:12.879130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:12.879466 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:12.879624 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:12.879880 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 20:20:12.880040 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 20:20:12.880941 22198 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59114
I20260924 20:20:12.881913 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22260
I20260924 20:20:12.888754 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22394
I20260924 20:20:12.893777 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22528
I20260924 20:20:12.898959 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22166
2026-09-24T20:20:12Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2159 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22669](info): set up 2 sockets
Sep 24 20:20:12 dist-test-slave-m1qq krb5kdc[22669](info): commencing operation
krb5kdc: starting...
W20260924 20:20:14.010927 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:14.146972 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41185
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:14.262555 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:14.262892 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:14.262969 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:14.267148 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:14.267256 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:14.267293 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:14.267331 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:14.267364 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:14.272902 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:14.274158 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:14.275300 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:14.284343 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:14.284366 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:14.284484 22682 server_base.cc:1034] running on GCE node
W20260924 20:20:14.284775 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:14.285398 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:14.286558 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:14.287737 22682 hybrid_clock.cc:648] HybridClock initialized: now 1790281214287702 us; error 50 us; skew 500 ppm
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:14.290750 22682 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:14.291855 22682 webserver.cc:460] Webserver started at http://127.18.172.126:45493/ using document root <none> and password file <none>
I20260924 20:20:14.292452 22682 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:14.292517 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:14.292727 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:14.294847 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "636d345913df463d8f50627d3faea860"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "1c80c529023c367483f329d1ee28a4e2"
server_key_iv: "400ec1a8379dfe57e8a675b25ff93a78"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.295483 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "636d345913df463d8f50627d3faea860"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "1c80c529023c367483f329d1ee28a4e2"
server_key_iv: "400ec1a8379dfe57e8a675b25ff93a78"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.299317 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:20:14.302232 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.303761 22682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:14.303913 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "636d345913df463d8f50627d3faea860"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "1c80c529023c367483f329d1ee28a4e2"
server_key_iv: "400ec1a8379dfe57e8a675b25ff93a78"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.304040 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:14.323633 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:14.326686 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:14.326898 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:14.334233 22682 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41185
I20260924 20:20:14.334259 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41185 every 8 connection(s)
I20260924 20:20:14.335260 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:14.338420 22750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:14.343834 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22682
I20260924 20:20:14.343989 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:14.344175 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: Bootstrap starting.
I20260924 20:20:14.344301 19121 external_mini_cluster.cc:1428] Setting key 36aaef0328161c5ea9d903fbc4028ec8
I20260924 20:20:14.346989 22750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:14.347918 22750 log.cc:824] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:14.350332 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: No bootstrap required, opened a new log
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:14.354574 22750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } }
I20260924 20:20:14.354841 22750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:14.354900 22750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 636d345913df463d8f50627d3faea860, State: Initialized, Role: FOLLOWER
I20260924 20:20:14.355434 22750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } }
I20260924 20:20:14.355628 22750 raft_consensus.cc:388] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:14.355713 22750 raft_consensus.cc:482] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:14.355826 22750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:14.357072 22750 raft_consensus.cc:504] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } }
I20260924 20:20:14.357448 22750 leader_election.cc:302] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 636d345913df463d8f50627d3faea860; no voters:
I20260924 20:20:14.357825 22750 leader_election.cc:288] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:14.358027 22755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:14.358386 22755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [term 1 LEADER]: Becoming Leader. State: Replica: 636d345913df463d8f50627d3faea860, State: Running, Role: LEADER
I20260924 20:20:14.358737 22755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } }
I20260924 20:20:14.359236 22750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:14.360597 22756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "636d345913df463d8f50627d3faea860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } } }
I20260924 20:20:14.360771 22756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:14.361150 22757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 636d345913df463d8f50627d3faea860. Latest consensus state: current_term: 1 leader_uuid: "636d345913df463d8f50627d3faea860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "636d345913df463d8f50627d3faea860" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41185 } } }
I20260924 20:20:14.361250 22757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:14.361276 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:14.345850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.18.172.126:41185)
0924 20:20:14.346317 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:14.346336 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:14.346379 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:20:14.347297 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:14.347323 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:14.347758 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:14.348142 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:14.349113 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.350553 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.351468 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.352164 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.352326 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:14.354885 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:14.354893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:14.354911 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:14.354958 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:14.357082 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.357589 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.357595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.357601 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.357773 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.358204 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.358211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.358214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.358915 (+ 701us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:14.359175 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:14.359483 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 20:20:14.359805 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":121,"threads_started":1}
W20260924 20:20:14.364336 22770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:20:14.364435 22770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:20:14.364518 22765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:14.365201 22765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:14.372080 22765 catalog_manager.cc:1347] Generated new cluster ID: abec7bfdc715452a87d7ed178e47fd62
I20260924 20:20:14.372203 22765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:14.387980 22765 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:14.389340 22765 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:14.399135 22765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 636d345913df463d8f50627d3faea860: Generated new TSK 0
I20260924 20:20:14.400113 22765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:14.442369 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:14.557471 22776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:14.557811 22776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:14.557886 22776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:14.562216 22776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:14.562311 22776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:14.562405 22776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:14.567746 22776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:14.568858 22776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:14.569902 22776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:14.579978 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:14.579978 22784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:14.580143 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:14.580156 22776 server_base.cc:1034] running on GCE node
I20260924 20:20:14.580811 22776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:14.581462 22776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:14.582655 22776 hybrid_clock.cc:648] HybridClock initialized: now 1790281214582620 us; error 47 us; skew 500 ppm
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:14.585757 22776 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:14.586660 22776 webserver.cc:460] Webserver started at http://127.18.172.65:35347/ using document root <none> and password file <none>
I20260924 20:20:14.587280 22776 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:14.587344 22776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:14.587595 22776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:14.589771 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "f12b5f8bf888415595f6b7209ea89f0d"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "da9792fc3f64841276c0900fbb5e4d4c"
server_key_iv: "9b4e4164decbc459759e798432b53b8f"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.590339 22776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "f12b5f8bf888415595f6b7209ea89f0d"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "da9792fc3f64841276c0900fbb5e4d4c"
server_key_iv: "9b4e4164decbc459759e798432b53b8f"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.594127 22776 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:14.596704 22791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.597999 22776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:14.598172 22776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "f12b5f8bf888415595f6b7209ea89f0d"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "da9792fc3f64841276c0900fbb5e4d4c"
server_key_iv: "9b4e4164decbc459759e798432b53b8f"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.598297 22776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:14.615897 22776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:14.619068 22776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:14.619239 22776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:14.619781 22776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:14.620702 22776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:14.620783 22776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.620872 22776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:14.620904 22776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.629856 22776 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:35297
I20260924 20:20:14.629874 22904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:35297 every 8 connection(s)
I20260924 20:20:14.630779 22776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:14.639330 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22776
I20260924 20:20:14.639485 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:14.639853 19121 external_mini_cluster.cc:1428] Setting key f0bdb8d6154eae385ceaba2591746766
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:14.645967 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:14.632640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:56285 (local address 127.18.172.126:41185)
0924 20:20:14.632795 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:14.632800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:14.633724 (+ 924us) server_negotiation.cc:407] Connection header received
0924 20:20:14.635130 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:14.635133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:14.635181 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:14.635255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:14.636483 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.637019 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.638153 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.638382 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.638465 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:14.641716 (+ 3251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:14.641720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:14.641723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:14.641758 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:14.643473 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.644153 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.644156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.644157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.644235 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.644685 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.644688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.644689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.644877 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:14.644984 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:14.645701 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 20:20:14.645814 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:20:14.647305 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:14.633091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41185 (local address 127.18.172.65:56285)
0924 20:20:14.633562 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:14.633609 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:14.634863 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:14.635423 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:14.635430 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:14.635829 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:14.636316 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:14.636329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.637131 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.637134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:14.638021 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:14.638030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.638516 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.638519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:14.638684 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:14.639339 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:14.639364 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:14.641534 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:14.643617 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:14.643624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:14.643643 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:14.643947 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:14.644336 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:14.644343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:14.644348 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:14.644561 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:14.645014 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:14.645024 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:14.645378 (+ 354us) client_negotiation.cc:769] Sending connection context
0924 20:20:14.645614 (+ 236us) client_negotiation.cc:240] Negotiation successful
0924 20:20:14.646031 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
I20260924 20:20:14.648675 22905 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41185
I20260924 20:20:14.648932 22905 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:14.649430 22905 heartbeater.cc:511] Master 127.18.172.126:41185 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:14.651055 22714 ts_manager.cc:194] Registered new tserver with Master: f12b5f8bf888415595f6b7209ea89f0d (127.18.172.65:35297)
I20260924 20:20:14.652288 22714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:56285
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:14.670272 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:14.782474 22910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:14.782744 22910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:14.782786 22910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:14.786942 22910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:14.787053 22910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:14.787168 22910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:14.792475 22910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:14.793510 22910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:14.794524 22910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:14.804342 22915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:14.804399 22910 server_base.cc:1034] running on GCE node
W20260924 20:20:14.804338 22918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:14.804358 22916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:14.805047 22910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:14.805682 22910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:14.806881 22910 hybrid_clock.cc:648] HybridClock initialized: now 1790281214806852 us; error 40 us; skew 500 ppm
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:14.809942 22910 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:14.810848 22910 webserver.cc:460] Webserver started at http://127.18.172.66:39515/ using document root <none> and password file <none>
I20260924 20:20:14.811460 22910 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:14.811548 22910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:14.811775 22910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:14.813803 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "4e78b37b5b9c4a9ca7d8ad924d86408a"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "e12df5a4d44b574274d340092dea1693"
server_key_iv: "c8fd920cf7f188e10f42844307b25de7"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.814366 22910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "4e78b37b5b9c4a9ca7d8ad924d86408a"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "e12df5a4d44b574274d340092dea1693"
server_key_iv: "c8fd920cf7f188e10f42844307b25de7"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.818095 22910 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:14.820657 22925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.821885 22910 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:14.822026 22910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "4e78b37b5b9c4a9ca7d8ad924d86408a"
format_stamp: "Formatted at 2026-09-24 20:20:14 on dist-test-slave-m1qq"
server_key: "e12df5a4d44b574274d340092dea1693"
server_key_iv: "c8fd920cf7f188e10f42844307b25de7"
server_key_version: "encryptionkey@0"
I20260924 20:20:14.822145 22910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:14.842137 22910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:14.845500 22910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:14.845715 22910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:14.846323 22910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:14.847167 22910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:14.847265 22910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.847363 22910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:14.847417 22910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:14.856122 22910 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:43111
I20260924 20:20:14.856204 23038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:43111 every 8 connection(s)
I20260924 20:20:14.857059 22910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:20:14 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281214, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:14.867110 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 22910
I20260924 20:20:14.867231 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:14.867515 19121 external_mini_cluster.cc:1428] Setting key cb07df8efe617d685ef96a2307c03cb9
I20260924 20:20:14.871536 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:14.858935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:55141 (local address 127.18.172.126:41185)
0924 20:20:14.859175 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:14.859179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:14.860156 (+ 977us) server_negotiation.cc:407] Connection header received
0924 20:20:14.861807 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:14.861810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:14.861856 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:14.861931 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:14.863065 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.863593 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.864698 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.864910 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.864994 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:14.867632 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:14.867636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:14.867639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:14.867672 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:14.869312 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.869819 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.869825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.869830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.869925 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:14.870267 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:14.870273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:14.870278 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:14.870533 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:14.870657 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:14.871320 (+ 663us) server_negotiation.cc:299] Negotiation successful
0924 20:20:14.871419 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260924 20:20:14.872820 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:14.859369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41185 (local address 127.18.172.66:55141)
0924 20:20:14.859967 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:14.860032 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:14.861558 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:14.862068 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:14.862074 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:14.862416 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:14.862886 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:14.862899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.863698 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.863700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:14.864589 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:14.864598 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:14.865039 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:14.865066 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:14.865201 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:14.865824 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:14.865840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:14.867465 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:14.869431 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:14.869435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:14.869448 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:14.869717 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:14.870016 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:14.870018 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:14.870020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:14.870176 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:14.870655 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:14.870664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:14.871035 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 20:20:14.871264 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 20:20:14.871661 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":121,"threads_started":1}
I20260924 20:20:14.874471 23039 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41185
I20260924 20:20:14.874836 23039 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:14.875502 23039 heartbeater.cc:511] Master 127.18.172.126:41185 requested a full tablet report, sending...
I20260924 20:20:14.877084 22714 ts_manager.cc:194] Registered new tserver with Master: 4e78b37b5b9c4a9ca7d8ad924d86408a (127.18.172.66:43111)
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:14.878485 22714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:55141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:14.899482 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:15.013965 23044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:15.014240 23044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:15.014285 23044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:15.018466 23044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:15.018537 23044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:15.018599 23044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:15.024466 23044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:15.025728 23044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:15.026790 23044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:15.036435 23049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:15.036612 23044 server_base.cc:1034] running on GCE node
W20260924 20:20:15.036520 23050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:15.036520 23052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:15.037213 23044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:15.037909 23044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:15.039080 23044 hybrid_clock.cc:648] HybridClock initialized: now 1790281215039049 us; error 50 us; skew 500 ppm
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281215, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:15.041851 23044 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:15.042711 23044 webserver.cc:460] Webserver started at http://127.18.172.67:45663/ using document root <none> and password file <none>
I20260924 20:20:15.043285 23044 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:15.043346 23044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:15.043517 23044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:15.045431 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "2e7349e6b2304ee9a195672dc88828f4"
format_stamp: "Formatted at 2026-09-24 20:20:15 on dist-test-slave-m1qq"
server_key: "9fb3f03ee2ccc404db39dff932c050d1"
server_key_iv: "35e312b14b1ee01d9993bdb4badd8686"
server_key_version: "encryptionkey@0"
I20260924 20:20:15.046048 23044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "2e7349e6b2304ee9a195672dc88828f4"
format_stamp: "Formatted at 2026-09-24 20:20:15 on dist-test-slave-m1qq"
server_key: "9fb3f03ee2ccc404db39dff932c050d1"
server_key_iv: "35e312b14b1ee01d9993bdb4badd8686"
server_key_version: "encryptionkey@0"
I20260924 20:20:15.049708 23044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 20:20:15.052106 23059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:15.053381 23044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:15.053524 23044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "2e7349e6b2304ee9a195672dc88828f4"
format_stamp: "Formatted at 2026-09-24 20:20:15 on dist-test-slave-m1qq"
server_key: "9fb3f03ee2ccc404db39dff932c050d1"
server_key_iv: "35e312b14b1ee01d9993bdb4badd8686"
server_key_version: "encryptionkey@0"
I20260924 20:20:15.053617 23044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:15.065261 23044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:15.068647 23044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:15.068799 23044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:15.069310 23044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:15.070176 23044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:15.070221 23044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:15.070261 23044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:15.070276 23044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:15.080024 23044 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:34235
I20260924 20:20:15.080041 23172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:34235 every 8 connection(s)
I20260924 20:20:15.080926 23044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:15.086371 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23044
I20260924 20:20:15.086529 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:15.086863 19121 external_mini_cluster.cc:1428] Setting key b599da14c8e6ee2ef113f5d318ea7afb
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281215, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:15.094928 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:15.082736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:58999 (local address 127.18.172.126:41185)
0924 20:20:15.082878 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:15.082882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:15.083751 (+ 869us) server_negotiation.cc:407] Connection header received
0924 20:20:15.085130 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:15.085133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:15.085191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:15.085290 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:15.086479 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.087185 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.088684 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.089006 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.089089 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:15.091502 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:15.091505 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:15.091507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:15.091532 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:15.093074 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.093536 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.093538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.093539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.093614 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.093964 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.093966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.093967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.094146 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:15.094222 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:15.094708 (+ 486us) server_negotiation.cc:299] Negotiation successful
0924 20:20:15.094805 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:20:15.095904 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:15.083175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41185 (local address 127.18.172.67:58999)
0924 20:20:15.083608 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:15.083651 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:15.084909 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:15.085469 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:15.085478 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:15.085851 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:15.086323 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:15.086336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.087340 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.087343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:15.088568 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:15.088578 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.089140 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.089143 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:15.089287 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:15.089868 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:15.089883 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:15.091341 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:15.093183 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:15.093188 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:15.093201 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:15.093447 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:15.093748 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:15.093750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:15.093752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:15.093885 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:15.094233 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:15.094238 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:15.094472 (+ 234us) client_negotiation.cc:769] Sending connection context
0924 20:20:15.094643 (+ 171us) client_negotiation.cc:240] Negotiation successful
0924 20:20:15.094904 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260924 20:20:15.097277 23173 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41185
I20260924 20:20:15.097560 23173 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:15.098088 23173 heartbeater.cc:511] Master 127.18.172.126:41185 requested a full tablet report, sending...
I20260924 20:20:15.099182 22714 ts_manager.cc:194] Registered new tserver with Master: 2e7349e6b2304ee9a195672dc88828f4 (127.18.172.67:34235)
I20260924 20:20:15.100037 22714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:58999
I20260924 20:20:15.100651 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[22669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:20:15.127473 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:15.117579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.18.172.65:35297)
0924 20:20:15.117900 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:15.117914 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:15.117937 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:20:15.117987 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:15.117992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:15.118192 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:15.118311 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:15.118998 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.119836 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.120785 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.121205 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.121344 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:15.123927 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:15.123936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:15.123950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:15.123974 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:15.125604 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.126055 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.126066 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.126067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.126166 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.126437 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.126441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.126443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.126691 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:15.126932 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:15.127121 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:20:15.127277 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":137,"threads_started":1}
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[22669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:15.138820 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:15.130506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.18.172.126:41185)
0924 20:20:15.130638 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:15.130643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:15.130709 (+ 66us) server_negotiation.cc:407] Connection header received
0924 20:20:15.130853 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:15.130856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:15.130896 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:15.130983 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:15.131609 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.132131 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.132930 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:15.133160 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:15.133270 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:15.135643 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:15.135647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:15.135650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:15.135684 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:15.137090 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.137550 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.137555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.137557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.137643 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:15.137915 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:15.137920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:15.137921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:15.138154 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:15.138298 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:15.138531 (+ 233us) server_negotiation.cc:299] Negotiation successful
0924 20:20:15.138650 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260924 20:20:15.139253 22714 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39862
I20260924 20:20:15.139886 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22776
I20260924 20:20:15.145212 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22910
I20260924 20:20:15.150341 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23044
I20260924 20:20:15.155148 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 22682
2026-09-24T20:20:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2257 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[23185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[23185](info): set up 2 sockets
Sep 24 20:20:15 dist-test-slave-m1qq krb5kdc[23185](info): commencing operation
krb5kdc: starting...
W20260924 20:20:16.208262 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:16.338223 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36233
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:36233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:16.448621 23198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:16.448907 23198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:16.448951 23198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:16.452952 23198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:16.453020 23198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:16.453040 23198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:16.453059 23198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:16.453075 23198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:16.458477 23198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:36233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:16.459527 23198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:16.460494 23198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:16.468881 23203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:16.468888 23204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:16.468888 23206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:16.469244 23198 server_base.cc:1034] running on GCE node
I20260924 20:20:16.469800 23198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:16.470928 23198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:16.472153 23198 hybrid_clock.cc:648] HybridClock initialized: now 1790281216472139 us; error 25 us; skew 500 ppm
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:16.478398 23198 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:16.479301 23198 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:16.479400 23198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:16.479640 23198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:16.481820 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "c116b89f76bc4ed696b66b00756c4909"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "9f54d47126163f5b8e9cea4e49b2df6e"
server_key_iv: "f51f2547aa18a8b0373646b9592789ca"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.482411 23198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "c116b89f76bc4ed696b66b00756c4909"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "9f54d47126163f5b8e9cea4e49b2df6e"
server_key_iv: "f51f2547aa18a8b0373646b9592789ca"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.486311 23198 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:16.488785 23212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.490091 23198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:20:16.490247 23198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "c116b89f76bc4ed696b66b00756c4909"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "9f54d47126163f5b8e9cea4e49b2df6e"
server_key_iv: "f51f2547aa18a8b0373646b9592789ca"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.490373 23198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:16.501127 23198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:16.504392 23198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:16.504606 23198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:16.511351 23198 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:36233
I20260924 20:20:16.511613 23264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:36233 every 8 connection(s)
I20260924 20:20:16.512099 23198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:16.514824 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23198
I20260924 20:20:16.514983 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:16.515278 23265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:16.515348 19121 external_mini_cluster.cc:1428] Setting key b57efe5b0c3c1571a4b6c0646398f544
I20260924 20:20:16.522552 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: Bootstrap starting.
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:16.525478 23265 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:16.526332 23265 log.cc:824] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:16.528286 23265 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: No bootstrap required, opened a new log
I20260924 20:20:16.530124 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:16.516741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55568 (local address 127.18.172.126:36233)
0924 20:20:16.517180 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:16.517195 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:16.517238 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:20:16.518056 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:16.518074 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:16.518408 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:16.518786 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:16.519733 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.520742 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.521920 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.522389 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.522678 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:16.525549 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:16.525554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:16.525567 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:16.525620 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:16.527354 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:16.527829 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:16.527834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:16.527847 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:16.527967 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:16.528252 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:16.528257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:16.528259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:16.528688 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:16.528893 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:16.529176 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 20:20:16.529443 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":148,"threads_started":1}
I20260924 20:20:16.532251 23265 raft_consensus.cc:348] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } }
I20260924 20:20:16.532599 23265 raft_consensus.cc:374] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:16.532663 23265 raft_consensus.cc:729] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c116b89f76bc4ed696b66b00756c4909, State: Initialized, Role: FOLLOWER
I20260924 20:20:16.533070 23265 consensus_queue.cc:260] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } }
I20260924 20:20:16.533212 23265 raft_consensus.cc:388] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:16.533272 23265 raft_consensus.cc:482] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:16.533380 23265 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:16.534333 23265 raft_consensus.cc:504] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } }
I20260924 20:20:16.534709 23265 leader_election.cc:302] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c116b89f76bc4ed696b66b00756c4909; no voters:
I20260924 20:20:16.535064 23265 leader_election.cc:288] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:16.535143 23270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:16.535331 23270 raft_consensus.cc:686] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [term 1 LEADER]: Becoming Leader. State: Replica: c116b89f76bc4ed696b66b00756c4909, State: Running, Role: LEADER
I20260924 20:20:16.535615 23270 consensus_queue.cc:237] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } }
I20260924 20:20:16.536357 23265 sys_catalog.cc:564] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:16.537257 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c116b89f76bc4ed696b66b00756c4909. Latest consensus state: current_term: 1 leader_uuid: "c116b89f76bc4ed696b66b00756c4909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } } }
I20260924 20:20:16.537407 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:16.537851 23279 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:16.538076 23271 sys_catalog.cc:455] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c116b89f76bc4ed696b66b00756c4909" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c116b89f76bc4ed696b66b00756c4909" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36233 } } }
I20260924 20:20:16.538170 23271 sys_catalog.cc:458] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:16.541051 23279 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:16.546891 23279 catalog_manager.cc:1347] Generated new cluster ID: 5d592957816041cd99965603efb8b26c
I20260924 20:20:16.547024 23279 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:16.566082 23279 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:16.567224 23279 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:16.579838 23279 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: Generated new TSK 0
I20260924 20:20:16.580757 23279 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:16.610580 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:16.723064 23291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:16.723402 23291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:16.723484 23291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:16.727681 23291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:16.727773 23291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:16.727870 23291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:16.733356 23291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:16.734547 23291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:16.735709 23291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:16.746140 23297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:16.746121 23296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:16.746121 23299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:16.746498 23291 server_base.cc:1034] running on GCE node
I20260924 20:20:16.747104 23291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:16.747756 23291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:16.748919 23291 hybrid_clock.cc:648] HybridClock initialized: now 1790281216748879 us; error 52 us; skew 500 ppm
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:16.751922 23291 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:16.752646 23291 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:16.752710 23291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:16.752879 23291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:16.754914 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "d83237185c7047629f9e756bfc9624bd"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "26c6581b277b1852c97c403344cf68e0"
server_key_iv: "e7e09bc70f71adcff0dfd44d723274ee"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.755525 23291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "d83237185c7047629f9e756bfc9624bd"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "26c6581b277b1852c97c403344cf68e0"
server_key_iv: "e7e09bc70f71adcff0dfd44d723274ee"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.759305 23291 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:20:16.761869 23305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.763509 23291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:16.763645 23291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "d83237185c7047629f9e756bfc9624bd"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "26c6581b277b1852c97c403344cf68e0"
server_key_iv: "e7e09bc70f71adcff0dfd44d723274ee"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.763759 23291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:16.780237 23291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:16.783674 23291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:16.783838 23291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:16.784396 23291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:16.785234 23291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:16.785282 23291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.785324 23291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:16.785341 23291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.794205 23291 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:45387
I20260924 20:20:16.794281 23418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:45387 every 8 connection(s)
I20260924 20:20:16.794885 23291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:16.798355 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23291
I20260924 20:20:16.798496 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:16.798774 19121 external_mini_cluster.cc:1428] Setting key 0cec72310d513278e3566a196ee542ca
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:16.809301 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:16.796688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:35681 (local address 127.18.172.126:36233)
0924 20:20:16.796829 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:16.796833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:16.797702 (+ 869us) server_negotiation.cc:407] Connection header received
0924 20:20:16.799238 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:16.799242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:16.799303 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:16.799407 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:16.800752 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.801296 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.802451 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.802703 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.802820 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:16.805443 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:16.805447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:16.805449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:16.805473 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:16.806942 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:16.807697 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:16.807699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:16.807701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:16.807777 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:16.808182 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:16.808184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:16.808185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:16.808370 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:16.808449 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:16.809013 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 20:20:16.809133 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:16.810181 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:16.797085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36233 (local address 127.18.172.65:35681)
0924 20:20:16.797551 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:16.797599 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:16.798942 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:16.799591 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:16.799600 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:16.800092 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:16.800608 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:16.800622 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.801424 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.801427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:16.802322 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:16.802332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:16.802862 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:16.802866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:16.803105 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:16.803725 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:16.803742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:16.805265 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:16.807153 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:16.807160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:16.807180 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:16.807579 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:16.807869 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:16.807874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:16.807877 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:16.808086 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:16.808495 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:16.808501 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:16.808782 (+ 281us) client_negotiation.cc:769] Sending connection context
0924 20:20:16.808951 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 20:20:16.809224 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:16.811448 23419 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36233
I20260924 20:20:16.811717 23419 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:16.812196 23419 heartbeater.cc:511] Master 127.18.172.126:36233 requested a full tablet report, sending...
I20260924 20:20:16.813725 23229 ts_manager.cc:194] Registered new tserver with Master: d83237185c7047629f9e756bfc9624bd (127.18.172.65:45387)
I20260924 20:20:16.814836 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:35681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:16.826735 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:16.941609 23424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:16.941864 23424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:16.941952 23424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:16.946030 23424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:16.946122 23424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:16.946218 23424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:16.951467 23424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:16.952508 23424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:16.953540 23424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:16.962873 23429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:16.962873 23430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:16.962976 23424 server_base.cc:1034] running on GCE node
W20260924 20:20:16.962873 23432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:16.963582 23424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:16.964249 23424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:16.965435 23424 hybrid_clock.cc:648] HybridClock initialized: now 1790281216965417 us; error 30 us; skew 500 ppm
Sep 24 20:20:16 dist-test-slave-m1qq krb5kdc[23185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:16.968283 23424 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:16.969025 23424 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:16.969118 23424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:16.969342 23424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:16.971380 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "c197e0138a984e5ea3fb20197dd162c1"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "064de96a7bfe25f5a7309bfce39222fe"
server_key_iv: "2fa76d5ff24c56122540718fe5f1ff83"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.971923 23424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "c197e0138a984e5ea3fb20197dd162c1"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "064de96a7bfe25f5a7309bfce39222fe"
server_key_iv: "2fa76d5ff24c56122540718fe5f1ff83"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.975471 23424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:16.977802 23438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.979128 23424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:16.979255 23424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "c197e0138a984e5ea3fb20197dd162c1"
format_stamp: "Formatted at 2026-09-24 20:20:16 on dist-test-slave-m1qq"
server_key: "064de96a7bfe25f5a7309bfce39222fe"
server_key_iv: "2fa76d5ff24c56122540718fe5f1ff83"
server_key_version: "encryptionkey@0"
I20260924 20:20:16.979364 23424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:16.989374 23424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:16.992359 23424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:16.992563 23424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:16.993125 23424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:16.993970 23424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:16.994046 23424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:16.994124 23424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:16.994167 23424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.002902 23424 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:44271
I20260924 20:20:17.002918 23551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:44271 every 8 connection(s)
I20260924 20:20:17.003517 23424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:20:17 dist-test-slave-m1qq krb5kdc[23185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281216, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:17.013250 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23424
I20260924 20:20:17.013481 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:17.013805 19121 external_mini_cluster.cc:1428] Setting key 2c67c34051d40fdf8d1ab1d6c9b808d4
I20260924 20:20:17.017690 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.005273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:35187 (local address 127.18.172.126:36233)
0924 20:20:17.005419 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.005422 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.006296 (+ 874us) server_negotiation.cc:407] Connection header received
0924 20:20:17.007667 (+ 1371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.007671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.007738 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.007833 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:17.009024 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.009571 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.010702 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.010933 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.011072 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.013598 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:17.013602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:17.013604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:17.013628 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:17.015325 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.015926 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.015929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.015930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.016014 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.016412 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.016414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.016415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.016585 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:17.016663 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.017336 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.017495 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:20:17.018640 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.005683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36233 (local address 127.18.172.66:35187)
0924 20:20:17.006146 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.006188 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.007405 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.008047 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.008054 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.008400 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:17.008872 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.008886 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.009697 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.009701 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.010575 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.010584 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.011108 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.011111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.011289 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.011895 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:17.011911 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:17.013421 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:17.015435 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:17.015441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:17.015457 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:17.015829 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:17.016107 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:17.016113 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:17.016118 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:17.016321 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:17.016690 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:17.016699 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:17.017042 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.017261 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.017601 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260924 20:20:17.020200 23552 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36233
I20260924 20:20:17.020469 23552 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:17.021016 23552 heartbeater.cc:511] Master 127.18.172.126:36233 requested a full tablet report, sending...
I20260924 20:20:17.022354 23229 ts_manager.cc:194] Registered new tserver with Master: c197e0138a984e5ea3fb20197dd162c1 (127.18.172.66:44271)
I20260924 20:20:17.023200 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:35187
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:17.044013 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:17.165912 23557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:17.166198 23557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:17.166293 23557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:17.171146 23557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:17.171213 23557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:17.171332 23557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:17.177287 23557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:17.178452 23557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:17.179597 23557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:17.189232 23563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:17.189188 23565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:17.189188 23562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:17.189558 23557 server_base.cc:1034] running on GCE node
I20260924 20:20:17.189965 23557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:17.190567 23557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:17.191730 23557 hybrid_clock.cc:648] HybridClock initialized: now 1790281217191702 us; error 37 us; skew 500 ppm
Sep 24 20:20:17 dist-test-slave-m1qq krb5kdc[23185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281217, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:17.194617 23557 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:17.195451 23557 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:17.195513 23557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:17.195731 23557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:17.197753 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "06baecf919e548b7a93ee42364693f1b"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "c059cdca31d2ef98ba4268914f81983d"
server_key_iv: "b4ff3c02cd741d7345d87fa95257fb91"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.198314 23557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "06baecf919e548b7a93ee42364693f1b"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "c059cdca31d2ef98ba4268914f81983d"
server_key_iv: "b4ff3c02cd741d7345d87fa95257fb91"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.202071 23557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:17.204509 23571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.205772 23557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:17.205907 23557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "06baecf919e548b7a93ee42364693f1b"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "c059cdca31d2ef98ba4268914f81983d"
server_key_iv: "b4ff3c02cd741d7345d87fa95257fb91"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.206017 23557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:17.215688 23557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:17.218458 23557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:17.218673 23557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:17.219290 23557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:17.220104 23557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:17.220180 23557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.220252 23557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:17.220287 23557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.229333 23557 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:45665
I20260924 20:20:17.229349 23684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:45665 every 8 connection(s)
I20260924 20:20:17.229939 23557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:17.230487 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23557
I20260924 20:20:17.230671 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:17.230970 19121 external_mini_cluster.cc:1428] Setting key ea73e7e01bf8c5b2906842bb65abb217
Sep 24 20:20:17 dist-test-slave-m1qq krb5kdc[23185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281217, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:17.244325 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.231864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:44213 (local address 127.18.172.126:36233)
0924 20:20:17.232034 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.232039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.233142 (+ 1103us) server_negotiation.cc:407] Connection header received
0924 20:20:17.234398 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.234401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.234445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.234515 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:17.235729 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.236357 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.237671 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.237974 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.238060 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.240550 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:17.240553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:17.240555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:17.240580 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:17.242052 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.242512 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.242514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.242516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.242587 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.242976 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.242978 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.242979 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.243216 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:17.243360 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.243968 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.244127 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:20:17.245368 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.232308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36233 (local address 127.18.172.67:44213)
0924 20:20:17.232863 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.232916 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.234212 (+ 1296us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.234656 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.234663 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.235101 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:17.235606 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.235620 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.236467 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.236470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.237545 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.237554 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.238117 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.238120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.238305 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.238860 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:17.238876 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:17.240388 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:17.242162 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:17.242167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:17.242180 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:17.242425 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:17.242704 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:17.242707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:17.242708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:17.242854 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:17.243354 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:17.243360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:17.243677 (+ 317us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.243880 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.244260 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":180,"threads_started":1}
I20260924 20:20:17.246481 23685 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36233
I20260924 20:20:17.246723 23685 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:17.247233 23685 heartbeater.cc:511] Master 127.18.172.126:36233 requested a full tablet report, sending...
I20260924 20:20:17.248188 23229 ts_manager.cc:194] Registered new tserver with Master: 06baecf919e548b7a93ee42364693f1b (127.18.172.67:45665)
I20260924 20:20:17.248768 23229 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:44213
I20260924 20:20:17.257264 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:20:17.266542 23268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.259248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.18.172.126:36233)
0924 20:20:17.259389 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.259394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.259630 (+ 236us) server_negotiation.cc:407] Connection header received
0924 20:20:17.259716 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.259718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.259759 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.259825 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:17.260580 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.261157 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.261976 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.262195 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.262272 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.263534 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:17.263537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:17.263539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:17.263561 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:17.264981 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.265350 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.265352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.265353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.265423 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:17.265663 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:17.265665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:17.265666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:17.265872 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:17.265978 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.266252 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.266345 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:20:17.269384 23229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:17.272423 23229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:17.287659 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.281307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43136 (local address 127.18.172.67:45665)
0924 20:20:17.281996 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.282011 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.282053 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:20:17.282139 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.282146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.282427 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.282594 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:17.283153 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284354 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.286165 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.286890 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287030 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287178 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.287373 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.287475 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":134,"threads_started":1}
I20260924 20:20:17.287657 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.281211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:45665 (local address 127.0.0.1:43136)
0924 20:20:17.281699 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.281749 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.281878 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.282783 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.282791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.282818 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:17.282973 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.282987 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284498 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.284503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.286047 (+ 1544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.286058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.287064 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.287175 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287212 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.287329 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.287466 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":42,"thread_start_us":111,"threads_started":1}
I20260924 20:20:17.287962 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.282171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.18.172.66:44271)
0924 20:20:17.282767 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.282777 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.282881 (+ 104us) server_negotiation.cc:407] Connection header received
0924 20:20:17.282968 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.282973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.283205 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.283361 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:17.283627 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284379 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.286382 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.286923 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287072 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287226 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.287692 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.287801 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":112,"threads_started":1}
I20260924 20:20:17.288252 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.281969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:44271 (local address 127.0.0.1:42056)
0924 20:20:17.282774 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.282788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.282874 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.283411 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.283414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.283432 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:17.283525 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.283531 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284535 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.284538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.286268 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.286275 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.287289 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.287457 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287477 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.287544 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.287603 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":559,"thread_start_us":87,"threads_started":1}
I20260924 20:20:17.288672 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.282391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58450 (local address 127.18.172.65:45387)
0924 20:20:17.282972 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.282977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.282996 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:17.283091 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.283095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.283245 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.283370 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:17.283762 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284840 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.286034 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.286886 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287036 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287178 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.287872 (+ 694us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.288526 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":135,"threads_started":1}
I20260924 20:20:17.288846 23699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.281296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45387 (local address 127.0.0.1:58450)
0924 20:20:17.282550 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.282563 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.282678 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.283523 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.283526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.283545 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:17.283637 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.283643 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.284963 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.284969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.285905 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.285914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.287254 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.287258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.287435 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.287465 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.287535 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.287609 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1185,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260924 20:20:17.290858 23486 tablet_service.cc:1467] Processing CreateTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 (DEFAULT_TABLE table=test-table [id=cd669567dcb8481f90af2fa1851fc62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:17.290926 23619 tablet_service.cc:1467] Processing CreateTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 (DEFAULT_TABLE table=test-table [id=cd669567dcb8481f90af2fa1851fc62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:17.291445 23353 tablet_service.cc:1467] Processing CreateTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 (DEFAULT_TABLE table=test-table [id=cd669567dcb8481f90af2fa1851fc62e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:17.292058 23486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd8be13133f4198a96d3ae52a6bba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:17.292071 23619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd8be13133f4198a96d3ae52a6bba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:17.292372 23353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbd8be13133f4198a96d3ae52a6bba32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:17.298146 23702 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: Bootstrap starting.
I20260924 20:20:17.299748 23704 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: Bootstrap starting.
I20260924 20:20:17.300194 23703 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Bootstrap starting.
I20260924 20:20:17.300552 23702 tablet_bootstrap.cc:654] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:17.301585 23702 log.cc:824] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:17.301868 23704 tablet_bootstrap.cc:654] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:17.302121 23703 tablet_bootstrap.cc:654] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:17.303182 23704 log.cc:824] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:17.303251 23703 log.cc:824] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:17.303439 23702 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: No bootstrap required, opened a new log
I20260924 20:20:17.303630 23702 ts_tablet_manager.cc:1397] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:20:17.305059 23704 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: No bootstrap required, opened a new log
I20260924 20:20:17.305230 23703 tablet_bootstrap.cc:492] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: No bootstrap required, opened a new log
I20260924 20:20:17.305231 23704 ts_tablet_manager.cc:1397] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:20:17.305356 23703 ts_tablet_manager.cc:1397] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:20:17.307516 23702 raft_consensus.cc:348] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.307798 23702 raft_consensus.cc:374] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:17.307863 23702 raft_consensus.cc:729] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c197e0138a984e5ea3fb20197dd162c1, State: Initialized, Role: FOLLOWER
I20260924 20:20:17.308202 23703 raft_consensus.cc:348] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.308342 23702 consensus_queue.cc:260] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.308430 23703 raft_consensus.cc:374] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:17.308485 23703 raft_consensus.cc:729] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06baecf919e548b7a93ee42364693f1b, State: Initialized, Role: FOLLOWER
I20260924 20:20:17.309049 23552 heartbeater.cc:503] Master 127.18.172.126:36233 was elected leader, sending a full tablet report...
I20260924 20:20:17.308929 23704 raft_consensus.cc:348] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.309144 23704 raft_consensus.cc:374] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:17.309191 23704 raft_consensus.cc:729] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83237185c7047629f9e756bfc9624bd, State: Initialized, Role: FOLLOWER
I20260924 20:20:17.309146 23703 consensus_queue.cc:260] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.309492 23702 ts_tablet_manager.cc:1428] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:20:17.309626 23704 consensus_queue.cc:260] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.310104 23703 ts_tablet_manager.cc:1428] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:20:17.310312 23419 heartbeater.cc:503] Master 127.18.172.126:36233 was elected leader, sending a full tablet report...
I20260924 20:20:17.310395 23685 heartbeater.cc:503] Master 127.18.172.126:36233 was elected leader, sending a full tablet report...
I20260924 20:20:17.310480 23704 ts_tablet_manager.cc:1428] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:20:17.417131 23710 raft_consensus.cc:482] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:17.417349 23710 raft_consensus.cc:504] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.418651 23710 leader_election.cc:288] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06baecf919e548b7a93ee42364693f1b (127.18.172.67:45665), c197e0138a984e5ea3fb20197dd162c1 (127.18.172.66:44271)
I20260924 20:20:17.422398 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.418841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:45665 (local address 127.18.172.65:46599)
0924 20:20:17.419199 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.419215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.419331 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.419611 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.419614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.419630 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:17.419757 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.419762 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.420482 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.420488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.421315 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.421324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.421958 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.421962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.422057 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.422080 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.422182 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.422251 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260924 20:20:17.422379 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.418999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46599 (local address 127.18.172.67:45665)
0924 20:20:17.419178 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.419183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.419285 (+ 102us) server_negotiation.cc:407] Connection header received
0924 20:20:17.419467 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.419470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.419511 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.419589 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:17.419875 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.420366 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.421448 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.421833 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.421963 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.422016 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.422192 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.422257 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:20:17.422556 23700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.419095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:50501 (local address 127.18.172.66:44271)
0924 20:20:17.419211 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.419217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.419447 (+ 230us) server_negotiation.cc:407] Connection header received
0924 20:20:17.419599 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.419602 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.419654 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.419737 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:17.419995 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.420528 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.421560 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.421960 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.422088 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.422139 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.422335 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.422405 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:20:17.422902 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.418994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:44271 (local address 127.18.172.65:50501)
0924 20:20:17.419347 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.419359 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.419485 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.419746 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.419749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.419764 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:17.419885 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.419890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.420644 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.420650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.421433 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:17.421442 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.422081 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.422084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:17.422205 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.422232 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.422321 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.422389 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":106,"threads_started":1}
I20260924 20:20:17.423040 23639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd8be13133f4198a96d3ae52a6bba32" candidate_uuid: "d83237185c7047629f9e756bfc9624bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baecf919e548b7a93ee42364693f1b" is_pre_election: true
I20260924 20:20:17.423393 23639 raft_consensus.cc:2454] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d83237185c7047629f9e756bfc9624bd in term 0.
I20260924 20:20:17.423558 23506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd8be13133f4198a96d3ae52a6bba32" candidate_uuid: "d83237185c7047629f9e756bfc9624bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c197e0138a984e5ea3fb20197dd162c1" is_pre_election: true
I20260924 20:20:17.423821 23506 raft_consensus.cc:2454] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d83237185c7047629f9e756bfc9624bd in term 0.
I20260924 20:20:17.423928 23309 leader_election.cc:302] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baecf919e548b7a93ee42364693f1b, d83237185c7047629f9e756bfc9624bd; no voters:
I20260924 20:20:17.424228 23710 raft_consensus.cc:2781] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:17.424320 23710 raft_consensus.cc:482] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:17.424372 23710 raft_consensus.cc:3037] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:17.425380 23710 raft_consensus.cc:504] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.425735 23710 leader_election.cc:288] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [CANDIDATE]: Term 1 election: Requested vote from peers 06baecf919e548b7a93ee42364693f1b (127.18.172.67:45665), c197e0138a984e5ea3fb20197dd162c1 (127.18.172.66:44271)
I20260924 20:20:17.426074 23639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd8be13133f4198a96d3ae52a6bba32" candidate_uuid: "d83237185c7047629f9e756bfc9624bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06baecf919e548b7a93ee42364693f1b"
I20260924 20:20:17.426146 23506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbd8be13133f4198a96d3ae52a6bba32" candidate_uuid: "d83237185c7047629f9e756bfc9624bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c197e0138a984e5ea3fb20197dd162c1"
I20260924 20:20:17.426213 23639 raft_consensus.cc:3037] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:17.426268 23506 raft_consensus.cc:3037] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:17.427410 23639 raft_consensus.cc:2454] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d83237185c7047629f9e756bfc9624bd in term 1.
I20260924 20:20:17.427538 23506 raft_consensus.cc:2454] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d83237185c7047629f9e756bfc9624bd in term 1.
I20260924 20:20:17.427762 23309 leader_election.cc:302] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06baecf919e548b7a93ee42364693f1b, d83237185c7047629f9e756bfc9624bd; no voters:
I20260924 20:20:17.427963 23710 raft_consensus.cc:2781] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:17.428189 23710 raft_consensus.cc:686] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 1 LEADER]: Becoming Leader. State: Replica: d83237185c7047629f9e756bfc9624bd, State: Running, Role: LEADER
I20260924 20:20:17.428537 23710 consensus_queue.cc:237] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } }
I20260924 20:20:17.431700 23228 catalog_manager.cc:5495] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd reported cstate change: term changed from 0 to 1, leader changed from <none> to d83237185c7047629f9e756bfc9624bd (127.18.172.65). New cstate: current_term: 1 leader_uuid: "d83237185c7047629f9e756bfc9624bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d83237185c7047629f9e756bfc9624bd" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 45387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:17.447861 23701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.444425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58454 (local address 127.18.172.65:45387)
0924 20:20:17.444554 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.444558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.444572 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:17.444685 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.444687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.444740 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.444829 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:17.445073 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.445598 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.446437 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:17.446669 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:17.446748 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:17.447054 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:17.447476 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:17.447571 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.447636 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.447683 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:17.453663 23506 raft_consensus.cc:1261] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 1 FOLLOWER]: Refusing update from remote peer d83237185c7047629f9e756bfc9624bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:17.453660 23639 raft_consensus.cc:1261] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 1 FOLLOWER]: Refusing update from remote peer d83237185c7047629f9e756bfc9624bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:17.454424 23713 consensus_queue.cc:1035] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "06baecf919e548b7a93ee42364693f1b" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:17.454638 23710 consensus_queue.cc:1035] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c197e0138a984e5ea3fb20197dd162c1" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 44271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:17.462724 23718 mvcc.cc:205] Tried to move back new op lower bound from 7332991866683256832 to 7332991866589822976. Current Snapshot: MvccSnapshot[applied={T|T < 7332991866683256832}]
I20260924 20:20:17.463773 23717 mvcc.cc:205] Tried to move back new op lower bound from 7332991866683256832 to 7332991866589822976. Current Snapshot: MvccSnapshot[applied={T|T < 7332991866683256832}]
I20260924 20:20:17.469944 23229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:17.470134 23229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:17.472414 23229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c116b89f76bc4ed696b66b00756c4909: Sending DeleteTablet for 3 replicas of tablet dbd8be13133f4198a96d3ae52a6bba32
I20260924 20:20:17.473245 23619 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:17 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:43136
I20260924 20:20:17.473398 23353 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:17 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:58450
I20260924 20:20:17.473754 23486 tablet_service.cc:1514] Processing DeleteTablet for tablet dbd8be13133f4198a96d3ae52a6bba32 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:17 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:42056
I20260924 20:20:17.473959 23728 tablet_replica.cc:330] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd: stopping tablet replica
I20260924 20:20:17.474186 23727 tablet_replica.cc:330] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: stopping tablet replica
I20260924 20:20:17.474352 23728 raft_consensus.cc:2229] T dbd8be13133f4198a96d3ae52a6bba32 P d83237185c7047629f9e756bfc9624bd [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:17.474465 23727 raft_consensus.cc:2229] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:17.474649 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23291
I20260924 20:20:17.475832 23727 raft_consensus.cc:2258] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:17.477433 23727 ts_tablet_manager.cc:1905] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:17.480849 23729 tablet_replica.cc:330] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1: stopping tablet replica
I20260924 20:20:17.480967 23727 ts_tablet_manager.cc:1918] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:17.481072 23727 log.cc:1196] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/dbd8be13133f4198a96d3ae52a6bba32
I20260924 20:20:17.481240 23729 raft_consensus.cc:2229] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:17.481424 23727 ts_tablet_manager.cc:1939] T dbd8be13133f4198a96d3ae52a6bba32 P 06baecf919e548b7a93ee42364693f1b: Deleting consensus metadata
I20260924 20:20:17.481633 23729 raft_consensus.cc:2258] T dbd8be13133f4198a96d3ae52a6bba32 P c197e0138a984e5ea3fb20197dd162c1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:20:17.482626 23215 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:45387 (error 108)
I20260924 20:20:17.482746 23216 catalog_manager.cc:4841] TS 06baecf919e548b7a93ee42364693f1b (127.18.172.67:45665): tablet dbd8be13133f4198a96d3ae52a6bba32 (table test-table [id=cd669567dcb8481f90af2fa1851fc62e]) successfully deleted
I20260924 20:20:17.483165 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23424
I20260924 20:20:17.484408 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.484054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:45387 (local address 127.0.0.1:58468)
0924 20:20:17.484223 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.484304 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:45387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 20:20:17.484694 23215 catalog_manager.cc:4584] TS d83237185c7047629f9e756bfc9624bd (127.18.172.65:45387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbd8be13133f4198a96d3ae52a6bba32: Network error: Client connection negotiation failed: client connection to 127.18.172.65:45387: connect: Connection refused (error 111)
I20260924 20:20:17.488889 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23557
I20260924 20:20:17.489936 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.489085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:44271 (local address 127.0.0.1:42058)
0924 20:20:17.489798 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.489858 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:44271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612}
I20260924 20:20:17.494840 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23198
2026-09-24T20:20:17Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2337 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T20:20:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:17Z Disabled control of system clock
I20260924 20:20:17.524668 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44781
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:20:17.635345 23734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:17.635632 23734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:17.635725 23734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:17.639860 23734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:17.639911 23734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:17.639926 23734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:17.639943 23734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:17.639981 23734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:17.644797 23734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:17.645901 23734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:17.647338 23734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:17.655889 23742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:17.655869 23734 server_base.cc:1034] running on GCE node
W20260924 20:20:17.655812 23739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:17.655833 23740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:17.656734 23734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:17.657810 23734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:17.658972 23734 hybrid_clock.cc:648] HybridClock initialized: now 1790281217658952 us; error 35 us; skew 500 ppm
I20260924 20:20:17.660859 23734 webserver.cc:460] Webserver started at http://127.18.172.126:40223/ using document root <none> and password file <none>
I20260924 20:20:17.661435 23734 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:17.661497 23734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:17.661733 23734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:17.663879 23734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "ac3951d52f4a4653ae3f3dda480ae3f2"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "fa5aa9e54f7eb0b87f7539813bb90525"
server_key_iv: "21c099b974ce84605a21d12100d3ac77"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.664448 23734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "ac3951d52f4a4653ae3f3dda480ae3f2"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "fa5aa9e54f7eb0b87f7539813bb90525"
server_key_iv: "21c099b974ce84605a21d12100d3ac77"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.668043 23734 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:17.670397 23748 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.671666 23734 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:17.671763 23734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "ac3951d52f4a4653ae3f3dda480ae3f2"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "fa5aa9e54f7eb0b87f7539813bb90525"
server_key_iv: "21c099b974ce84605a21d12100d3ac77"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.671839 23734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:17.676254 23734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:17.676803 23734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:17.676935 23734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:17.683977 23734 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44781
I20260924 20:20:17.683983 23800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44781 every 8 connection(s)
I20260924 20:20:17.684944 23734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:17.688089 23801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:17.690281 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23734
I20260924 20:20:17.690376 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:17.690670 19121 external_mini_cluster.cc:1428] Setting key d07083cf65549a92555f13ab11932f0f
I20260924 20:20:17.694093 23801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: Bootstrap starting.
I20260924 20:20:17.696430 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.692167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.18.172.126:44781)
0924 20:20:17.692921 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.692936 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.692972 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:20:17.693584 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.693612 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.693967 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.694339 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:17.695180 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:17.695185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:17.695197 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:17.695208 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:17.695262 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:17.695278 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:17.695423 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.695496 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.695756 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":154,"threads_started":1}
I20260924 20:20:17.696668 23801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:17.697485 23801 log.cc:824] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:17.699581 23801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: No bootstrap required, opened a new log
I20260924 20:20:17.703233 23801 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } }
I20260924 20:20:17.703459 23801 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:17.703492 23801 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3951d52f4a4653ae3f3dda480ae3f2, State: Initialized, Role: FOLLOWER
I20260924 20:20:17.703913 23801 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } }
I20260924 20:20:17.704035 23801 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:17.704089 23801 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:17.704160 23801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:17.705005 23801 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } }
I20260924 20:20:17.705377 23801 leader_election.cc:302] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac3951d52f4a4653ae3f3dda480ae3f2; no voters:
I20260924 20:20:17.705729 23801 leader_election.cc:288] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:17.705895 23806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:17.706131 23806 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [term 1 LEADER]: Becoming Leader. State: Replica: ac3951d52f4a4653ae3f3dda480ae3f2, State: Running, Role: LEADER
I20260924 20:20:17.706442 23806 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } }
I20260924 20:20:17.707070 23801 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:17.708187 23807 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } } }
I20260924 20:20:17.708197 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac3951d52f4a4653ae3f3dda480ae3f2. Latest consensus state: current_term: 1 leader_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac3951d52f4a4653ae3f3dda480ae3f2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44781 } } }
I20260924 20:20:17.708420 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:17.708359 23807 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:17.708822 23815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:17.711639 23815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:17.717033 23815 catalog_manager.cc:1347] Generated new cluster ID: e43ef75f42124f4cad387b2bd156f3a4
I20260924 20:20:17.717123 23815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:17.735349 23815 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:17.735675 23815 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:17.741840 23815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: Generated new TSK 0
I20260924 20:20:17.742511 23815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:17.752120 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--builtin_ntp_servers=127.18.172.84:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 20:20:17.870338 23825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:17.870651 23825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:17.870724 23825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:17.874866 23825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:17.874943 23825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:17.875072 23825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:17.880044 23825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:17.881124 23825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:17.882547 23825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:17.892033 23831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:17.892105 23833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:17.892122 23830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:17.892361 23825 server_base.cc:1034] running on GCE node
I20260924 20:20:17.892808 23825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:17.893505 23825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:17.894727 23825 hybrid_clock.cc:648] HybridClock initialized: now 1790281217894709 us; error 31 us; skew 500 ppm
I20260924 20:20:17.896806 23825 webserver.cc:460] Webserver started at http://127.18.172.65:34699/ using document root <none> and password file <none>
I20260924 20:20:17.897444 23825 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:17.897548 23825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:17.897800 23825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:17.900141 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "c1fbcacef970490b9017e917afc2c41f"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "207be9006d4472b885799ee53c361810"
server_key_iv: "af23a67a9539f8e68a4df9e7c647f819"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.900740 23825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "c1fbcacef970490b9017e917afc2c41f"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "207be9006d4472b885799ee53c361810"
server_key_iv: "af23a67a9539f8e68a4df9e7c647f819"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.904803 23825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:20:17.907644 23839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.909176 23825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:17.909332 23825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "c1fbcacef970490b9017e917afc2c41f"
format_stamp: "Formatted at 2026-09-24 20:20:17 on dist-test-slave-m1qq"
server_key: "207be9006d4472b885799ee53c361810"
server_key_iv: "af23a67a9539f8e68a4df9e7c647f819"
server_key_version: "encryptionkey@0"
I20260924 20:20:17.909474 23825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:17.914551 23825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:17.915269 23825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:17.915473 23825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:17.916038 23825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:17.917002 23825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:17.917074 23825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.917150 23825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:17.917207 23825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:17.926301 23825 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:34635
I20260924 20:20:17.926322 23952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:34635 every 8 connection(s)
I20260924 20:20:17.927287 23825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:17.928629 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23825
I20260924 20:20:17.928774 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:17.929122 19121 external_mini_cluster.cc:1428] Setting key 0a51c32a476e5892af53b4cf161c323a
I20260924 20:20:17.931329 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--builtin_ntp_servers=127.18.172.84:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 20:20:17.934193 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.929281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:58529 (local address 127.18.172.126:44781)
0924 20:20:17.929438 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:17.929442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:17.930771 (+ 1329us) server_negotiation.cc:407] Connection header received
0924 20:20:17.932336 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:17.932339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:17.932382 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:17.932475 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:17.933670 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:17.933674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:17.933677 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:17.933689 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:17.933714 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:17.933732 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:17.933833 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:17.934007 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 20:20:17.934053 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:20:17.935253 23955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:17.929784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44781 (local address 127.18.172.65:58529)
0924 20:20:17.930613 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:17.930677 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:17.932121 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:17.932666 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:17.932673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:17.932769 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:17.933395 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:17.933397 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:17.933441 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:17.933514 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:17.933829 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:17.933839 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:17.933848 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:20:17.934056 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 20:20:17.934310 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":136,"threads_started":1}
I20260924 20:20:17.936738 23953 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44781
I20260924 20:20:17.936995 23953 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:17.937422 23953 heartbeater.cc:511] Master 127.18.172.126:44781 requested a full tablet report, sending...
I20260924 20:20:17.939433 23765 ts_manager.cc:194] Registered new tserver with Master: c1fbcacef970490b9017e917afc2c41f (127.18.172.65:34635)
W20260924 20:20:18.043290 23956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:18.043550 23956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:18.043589 23956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:18.047706 23956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:18.047760 23956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:18.047817 23956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:18.052466 23956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:18.053463 23956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:18.054749 23956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:18.064070 23961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:18.064114 23964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:18.064137 23962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:18.064275 23956 server_base.cc:1034] running on GCE node
I20260924 20:20:18.064775 23956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:18.065423 23956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:18.066609 23956 hybrid_clock.cc:648] HybridClock initialized: now 1790281218066574 us; error 44 us; skew 500 ppm
I20260924 20:20:18.068404 23956 webserver.cc:460] Webserver started at http://127.18.172.66:38787/ using document root <none> and password file <none>
I20260924 20:20:18.068967 23956 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:18.069027 23956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:18.069247 23956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:18.071308 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "8d259dd780be4b21a0668aaa8a23a8c4"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "b793e9acb65afdf8bfb9f5b2772d813d"
server_key_iv: "72361a000a530e87a15c5cf1d51f7011"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.071843 23956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "8d259dd780be4b21a0668aaa8a23a8c4"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "b793e9acb65afdf8bfb9f5b2772d813d"
server_key_iv: "72361a000a530e87a15c5cf1d51f7011"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.075783 23956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:20:18.078354 23970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.079622 23956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:18.079754 23956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "8d259dd780be4b21a0668aaa8a23a8c4"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "b793e9acb65afdf8bfb9f5b2772d813d"
server_key_iv: "72361a000a530e87a15c5cf1d51f7011"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.079871 23956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:18.084385 23956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:18.084972 23956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:18.085158 23956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:18.085688 23956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:18.086556 23956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:18.086633 23956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.086727 23956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:18.086781 23956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.095659 23956 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:46281
I20260924 20:20:18.095685 24083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:46281 every 8 connection(s)
I20260924 20:20:18.096539 23956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:18.097249 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 23956
I20260924 20:20:18.097388 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:18.097687 19121 external_mini_cluster.cc:1428] Setting key 9db9c3869c70d7d29593df985d07ab17
I20260924 20:20:18.099972 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--builtin_ntp_servers=127.18.172.84:42049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 20:20:18.103071 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.098442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:41421 (local address 127.18.172.126:44781)
0924 20:20:18.098558 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.098564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.099871 (+ 1307us) server_negotiation.cc:407] Connection header received
0924 20:20:18.101324 (+ 1453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.101327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.101368 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.101460 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.102560 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.102563 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.102566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.102579 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.102601 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.102611 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.102714 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.102867 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.102914 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 20:20:18.103946 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.099004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44781 (local address 127.18.172.66:41421)
0924 20:20:18.099732 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.099776 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.101118 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.101624 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.101631 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.101714 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.102308 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.102309 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.102349 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.102415 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.102711 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.102719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.102727 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.102899 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.103169 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
I20260924 20:20:18.105113 24084 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44781
I20260924 20:20:18.105345 24084 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:18.105764 24084 heartbeater.cc:511] Master 127.18.172.126:44781 requested a full tablet report, sending...
I20260924 20:20:18.106813 23765 ts_manager.cc:194] Registered new tserver with Master: 8d259dd780be4b21a0668aaa8a23a8c4 (127.18.172.66:46281)
W20260924 20:20:18.211725 24087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:18.211989 24087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:18.212028 24087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:18.216078 24087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:18.216154 24087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:18.216264 24087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:18.221024 24087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:18.222090 24087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:18.223553 24087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:18.233346 24093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:18.233346 24092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:18.233481 24087 server_base.cc:1034] running on GCE node
W20260924 20:20:18.233405 24095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:18.234078 24087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:18.234757 24087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:18.235955 24087 hybrid_clock.cc:648] HybridClock initialized: now 1790281218235943 us; error 26 us; skew 500 ppm
I20260924 20:20:18.237941 24087 webserver.cc:460] Webserver started at http://127.18.172.67:41579/ using document root <none> and password file <none>
I20260924 20:20:18.238556 24087 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:18.238651 24087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:18.238905 24087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:18.241269 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "2ec4a26529524573a97e416657c8033d"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "4b6f5274bcb7fd9dc1885ab73f851617"
server_key_iv: "49494b6bfb05c492b90b8430a58660f0"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.241963 24087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "2ec4a26529524573a97e416657c8033d"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "4b6f5274bcb7fd9dc1885ab73f851617"
server_key_iv: "49494b6bfb05c492b90b8430a58660f0"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.246049 24087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:18.248842 24101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.250141 24087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:18.250283 24087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "2ec4a26529524573a97e416657c8033d"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "4b6f5274bcb7fd9dc1885ab73f851617"
server_key_iv: "49494b6bfb05c492b90b8430a58660f0"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.250404 24087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:18.255141 24087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:18.255800 24087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:18.255981 24087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:18.256527 24087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:18.257417 24087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:18.257463 24087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.257539 24087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:18.257577 24087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.266227 24087 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:43279
I20260924 20:20:18.266247 24214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:43279 every 8 connection(s)
I20260924 20:20:18.267200 24087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:18.273211 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.269037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:50931 (local address 127.18.172.126:44781)
0924 20:20:18.269185 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.269189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.270222 (+ 1033us) server_negotiation.cc:407] Connection header received
0924 20:20:18.271595 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.271599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.271644 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.271723 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.272725 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.272729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.272732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.272744 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.272766 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.272776 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.272883 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.273019 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.273070 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:18.274047 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.269435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44781 (local address 127.18.172.67:50931)
0924 20:20:18.270078 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.270121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.271394 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.271874 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.271881 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.271965 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.272471 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.272473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.272511 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.272573 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.272866 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.272874 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.272880 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.273056 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.273291 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":110,"threads_started":1}
I20260924 20:20:18.275202 24215 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44781
I20260924 20:20:18.275430 24215 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:18.275895 24215 heartbeater.cc:511] Master 127.18.172.126:44781 requested a full tablet report, sending...
I20260924 20:20:18.276130 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24087
I20260924 20:20:18.276245 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:18.276547 19121 external_mini_cluster.cc:1428] Setting key 6145785e969dd7b7eba2709d15af3c3d
I20260924 20:20:18.277156 23765 ts_manager.cc:194] Registered new tserver with Master: 2ec4a26529524573a97e416657c8033d (127.18.172.67:43279)
I20260924 20:20:18.278424 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:20:18.281854 23804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.280376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.18.172.126:44781)
0924 20:20:18.280535 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.280539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.280701 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:20:18.280796 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.280798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.280830 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.280896 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.281478 (+ 582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.281481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.281483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.281493 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.281511 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.281518 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.281603 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.281656 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.281688 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:20:18.285111 23765 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:18.288213 23765 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:18.299357 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.296237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:46281 (local address 127.0.0.1:43134)
0924 20:20:18.296655 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.296713 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.296818 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.297410 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.297417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.297495 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.298467 (+ 972us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.298469 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.298490 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.298566 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.298977 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.298985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.298991 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.299129 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.299202 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":120,"threads_started":1}
I20260924 20:20:18.299791 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.296412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43134 (local address 127.18.172.66:46281)
0924 20:20:18.296803 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.296815 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.296842 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:20:18.296906 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.296912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.297125 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.297268 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.298720 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.298728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.298743 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.298759 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.298802 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.298834 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.298934 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.299542 (+ 608us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.299630 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":160,"threads_started":1}
I20260924 20:20:18.300086 24230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.297356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34635 (local address 127.0.0.1:60300)
0924 20:20:18.297795 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.297809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.297879 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.298622 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.298628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.298648 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.299057 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.299059 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.299069 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.299109 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.299820 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.299825 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.299831 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.299891 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.299925 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":84,"threads_started":1}
I20260924 20:20:18.300297 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.297783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.18.172.65:34635)
0924 20:20:18.298198 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.298209 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.298231 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:20:18.298282 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.298287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.298468 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.298843 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.299630 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.299634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.299644 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.299653 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.299688 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.299710 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.300039 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.300083 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.300157 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":116,"threads_started":1}
I20260924 20:20:18.300565 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.297226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.18.172.67:43279)
0924 20:20:18.298142 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.298155 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.298179 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:20:18.298228 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.298232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.298374 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.298510 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.299371 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.299377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.299387 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.299405 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.299654 (+ 249us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.299689 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.299785 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.300308 (+ 523us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.300378 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":152,"threads_started":1}
I20260924 20:20:18.300776 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.297049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:43279 (local address 127.0.0.1:49954)
0924 20:20:18.298049 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.298065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.298131 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.298493 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.298496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.298511 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.299213 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.299217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.299232 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.299256 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.299829 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.299832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.299835 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.299913 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.299949 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":86,"threads_started":1}
I20260924 20:20:18.302892 23887 tablet_service.cc:1467] Processing CreateTablet for tablet ce8fcad8ff404834985cb9caabfacf25 (DEFAULT_TABLE table=test-table [id=cf537a74f5294155ad234d380368d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:18.302560 24018 tablet_service.cc:1467] Processing CreateTablet for tablet ce8fcad8ff404834985cb9caabfacf25 (DEFAULT_TABLE table=test-table [id=cf537a74f5294155ad234d380368d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:18.303282 24149 tablet_service.cc:1467] Processing CreateTablet for tablet ce8fcad8ff404834985cb9caabfacf25 (DEFAULT_TABLE table=test-table [id=cf537a74f5294155ad234d380368d78c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:18.304042 23887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8fcad8ff404834985cb9caabfacf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:18.304122 24018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8fcad8ff404834985cb9caabfacf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:18.304193 24149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce8fcad8ff404834985cb9caabfacf25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:18.310525 24232 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Bootstrap starting.
I20260924 20:20:18.311643 24233 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Bootstrap starting.
I20260924 20:20:18.313046 24232 tablet_bootstrap.cc:654] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:18.313556 24233 tablet_bootstrap.cc:654] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:18.313887 24232 log.cc:824] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:18.314327 24233 log.cc:824] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:18.314990 24234 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Bootstrap starting.
I20260924 20:20:18.315973 24233 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: No bootstrap required, opened a new log
I20260924 20:20:18.316120 24232 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: No bootstrap required, opened a new log
I20260924 20:20:18.316203 24233 ts_tablet_manager.cc:1397] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:20:18.316295 24232 ts_tablet_manager.cc:1397] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:20:18.317039 24234 tablet_bootstrap.cc:654] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:18.317842 24234 log.cc:824] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:18.319988 24234 tablet_bootstrap.cc:492] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: No bootstrap required, opened a new log
I20260924 20:20:18.320179 24234 ts_tablet_manager.cc:1397] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 20:20:18.320441 24233 raft_consensus.cc:348] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.320683 24233 raft_consensus.cc:374] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:18.320742 24233 raft_consensus.cc:729] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1fbcacef970490b9017e917afc2c41f, State: Initialized, Role: FOLLOWER
I20260924 20:20:18.320708 24232 raft_consensus.cc:348] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.320961 24232 raft_consensus.cc:374] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:18.321026 24232 raft_consensus.cc:729] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d259dd780be4b21a0668aaa8a23a8c4, State: Initialized, Role: FOLLOWER
I20260924 20:20:18.321273 24233 consensus_queue.cc:260] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.321444 24232 consensus_queue.cc:260] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.322110 24084 heartbeater.cc:503] Master 127.18.172.126:44781 was elected leader, sending a full tablet report...
I20260924 20:20:18.322314 24233 ts_tablet_manager.cc:1428] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260924 20:20:18.322372 24232 ts_tablet_manager.cc:1428] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:20:18.322607 23953 heartbeater.cc:503] Master 127.18.172.126:44781 was elected leader, sending a full tablet report...
I20260924 20:20:18.324422 24234 raft_consensus.cc:348] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.324724 24234 raft_consensus.cc:374] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:18.324786 24234 raft_consensus.cc:729] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec4a26529524573a97e416657c8033d, State: Initialized, Role: FOLLOWER
I20260924 20:20:18.325222 24234 consensus_queue.cc:260] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.325820 24215 heartbeater.cc:503] Master 127.18.172.126:44781 was elected leader, sending a full tablet report...
I20260924 20:20:18.326171 24234 ts_tablet_manager.cc:1428] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
W20260924 20:20:18.347497 24085 tablet.cc:2369] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:20:18.428637 23954 tablet.cc:2369] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:18.496855 24238 raft_consensus.cc:482] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:18.497109 24238 raft_consensus.cc:504] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.498323 24238 leader_election.cc:288] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d259dd780be4b21a0668aaa8a23a8c4 (127.18.172.66:46281), 2ec4a26529524573a97e416657c8033d (127.18.172.67:43279)
I20260924 20:20:18.500682 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.498790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:34255 (local address 127.18.172.66:46281)
0924 20:20:18.498934 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.498940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.499070 (+ 130us) server_negotiation.cc:407] Connection header received
0924 20:20:18.499264 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.499267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.499308 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.499365 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.500202 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.500206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.500210 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.500226 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.500242 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.500250 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.500349 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.500469 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.500516 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:18.500705 24241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.498627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:46281 (local address 127.18.172.65:34255)
0924 20:20:18.498934 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.498951 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.499134 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.499494 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.499496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.499511 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.500042 (+ 531us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.500044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.500053 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.500077 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.500363 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.500366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.500369 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.500443 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.500494 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":111,"threads_started":1}
I20260924 20:20:18.500746 24229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.499204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:44399 (local address 127.18.172.67:43279)
0924 20:20:18.499331 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.499337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.499404 (+ 67us) server_negotiation.cc:407] Connection header received
0924 20:20:18.499494 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.499496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.499530 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.499639 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.500348 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.500351 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.500355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.500368 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.500386 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.500394 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.500517 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.500593 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.500629 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:18.500926 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.499058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:43279 (local address 127.18.172.65:44399)
0924 20:20:18.499338 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.499350 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.499429 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.499657 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.499659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.499694 (+ 35us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:18.500203 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:18.500204 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:18.500213 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:18.500236 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:18.500514 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:18.500517 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:18.500520 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:20:18.500595 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:20:18.500623 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":102,"threads_started":1}
I20260924 20:20:18.501315 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8fcad8ff404834985cb9caabfacf25" candidate_uuid: "c1fbcacef970490b9017e917afc2c41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" is_pre_election: true
I20260924 20:20:18.501533 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8fcad8ff404834985cb9caabfacf25" candidate_uuid: "c1fbcacef970490b9017e917afc2c41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec4a26529524573a97e416657c8033d" is_pre_election: true
I20260924 20:20:18.501653 24038 raft_consensus.cc:2454] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fbcacef970490b9017e917afc2c41f in term 0.
I20260924 20:20:18.501776 24169 raft_consensus.cc:2454] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1fbcacef970490b9017e917afc2c41f in term 0.
I20260924 20:20:18.502166 23843 leader_election.cc:302] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d259dd780be4b21a0668aaa8a23a8c4, c1fbcacef970490b9017e917afc2c41f; no voters:
I20260924 20:20:18.502463 24238 raft_consensus.cc:2781] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:18.502542 24238 raft_consensus.cc:482] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:18.502571 24238 raft_consensus.cc:3037] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:18.503736 24238 raft_consensus.cc:504] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.504119 24238 leader_election.cc:288] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [CANDIDATE]: Term 1 election: Requested vote from peers 8d259dd780be4b21a0668aaa8a23a8c4 (127.18.172.66:46281), 2ec4a26529524573a97e416657c8033d (127.18.172.67:43279)
I20260924 20:20:18.504528 24038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8fcad8ff404834985cb9caabfacf25" candidate_uuid: "c1fbcacef970490b9017e917afc2c41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d259dd780be4b21a0668aaa8a23a8c4"
I20260924 20:20:18.504580 24169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce8fcad8ff404834985cb9caabfacf25" candidate_uuid: "c1fbcacef970490b9017e917afc2c41f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec4a26529524573a97e416657c8033d"
I20260924 20:20:18.504664 24038 raft_consensus.cc:3037] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:18.504706 24169 raft_consensus.cc:3037] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:18.505931 24038 raft_consensus.cc:2454] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fbcacef970490b9017e917afc2c41f in term 1.
I20260924 20:20:18.505931 24169 raft_consensus.cc:2454] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1fbcacef970490b9017e917afc2c41f in term 1.
I20260924 20:20:18.506218 23843 leader_election.cc:302] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d259dd780be4b21a0668aaa8a23a8c4, c1fbcacef970490b9017e917afc2c41f; no voters:
I20260924 20:20:18.506425 24238 raft_consensus.cc:2781] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:18.506675 24238 raft_consensus.cc:686] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [term 1 LEADER]: Becoming Leader. State: Replica: c1fbcacef970490b9017e917afc2c41f, State: Running, Role: LEADER
I20260924 20:20:18.507091 24238 consensus_queue.cc:237] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } }
I20260924 20:20:18.510459 23764 catalog_manager.cc:5495] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f reported cstate change: term changed from 0 to 1, leader changed from <none> to c1fbcacef970490b9017e917afc2c41f (127.18.172.65). New cstate: current_term: 1 leader_uuid: "c1fbcacef970490b9017e917afc2c41f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1fbcacef970490b9017e917afc2c41f" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34635 } health_report { overall_health: HEALTHY } } }
W20260924 20:20:18.518116 24216 tablet.cc:2369] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:18.534731 24231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.533315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.18.172.65:34635)
0924 20:20:18.533436 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.533439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.533451 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:18.533621 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.533623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.533685 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.533756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.534419 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.534422 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.534424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.534435 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.534452 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.534460 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.534574 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.534598 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.534629 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:20:18.540732 24169 raft_consensus.cc:1261] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 1 FOLLOWER]: Refusing update from remote peer c1fbcacef970490b9017e917afc2c41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:18.540758 24038 raft_consensus.cc:1261] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 1 FOLLOWER]: Refusing update from remote peer c1fbcacef970490b9017e917afc2c41f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:18.541457 24243 consensus_queue.cc:1035] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d259dd780be4b21a0668aaa8a23a8c4" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 46281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:18.541669 24238 consensus_queue.cc:1035] T ce8fcad8ff404834985cb9caabfacf25 P c1fbcacef970490b9017e917afc2c41f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec4a26529524573a97e416657c8033d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:18.552053 24248 mvcc.cc:205] Tried to move back new op lower bound from 7332991871135203328 to 7332991871007666176. Current Snapshot: MvccSnapshot[applied={T|T < 7332991871135203328}]
I20260924 20:20:18.557615 23764 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:18.557771 23764 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:18.560233 23764 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac3951d52f4a4653ae3f3dda480ae3f2: Sending DeleteTablet for 3 replicas of tablet ce8fcad8ff404834985cb9caabfacf25
I20260924 20:20:18.561062 24018 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8fcad8ff404834985cb9caabfacf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:18 UTC) from {username='slave'} at 127.0.0.1:43134
I20260924 20:20:18.561187 23887 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8fcad8ff404834985cb9caabfacf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:18 UTC) from {username='slave'} at 127.0.0.1:60300
I20260924 20:20:18.561686 24149 tablet_service.cc:1514] Processing DeleteTablet for tablet ce8fcad8ff404834985cb9caabfacf25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:18 UTC) from {username='slave'} at 127.0.0.1:49954
I20260924 20:20:18.561975 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23825
I20260924 20:20:18.562700 24256 tablet_replica.cc:330] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: stopping tablet replica
I20260924 20:20:18.563118 24256 raft_consensus.cc:2229] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:18.563481 24256 raft_consensus.cc:2258] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:18.564064 24258 tablet_replica.cc:330] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: stopping tablet replica
I20260924 20:20:18.564332 24258 raft_consensus.cc:2229] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:18.564644 24258 raft_consensus.cc:2258] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:18.565153 24256 ts_tablet_manager.cc:1905] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:18.565846 24258 ts_tablet_manager.cc:1905] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:18.567957 24258 ts_tablet_manager.cc:1918] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:18.568073 24258 log.cc:1196] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/ce8fcad8ff404834985cb9caabfacf25
I20260924 20:20:18.568342 24258 ts_tablet_manager.cc:1939] T ce8fcad8ff404834985cb9caabfacf25 P 2ec4a26529524573a97e416657c8033d: Deleting consensus metadata
I20260924 20:20:18.568645 24256 ts_tablet_manager.cc:1918] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:18.568748 24256 log.cc:1196] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/ce8fcad8ff404834985cb9caabfacf25
I20260924 20:20:18.569069 24256 ts_tablet_manager.cc:1939] T ce8fcad8ff404834985cb9caabfacf25 P 8d259dd780be4b21a0668aaa8a23a8c4: Deleting consensus metadata
I20260924 20:20:18.569583 23752 catalog_manager.cc:4841] TS 2ec4a26529524573a97e416657c8033d (127.18.172.67:43279): tablet ce8fcad8ff404834985cb9caabfacf25 (table test-table [id=cf537a74f5294155ad234d380368d78c]) successfully deleted
I20260924 20:20:18.570235 23752 catalog_manager.cc:4841] TS 8d259dd780be4b21a0668aaa8a23a8c4 (127.18.172.66:46281): tablet ce8fcad8ff404834985cb9caabfacf25 (table test-table [id=cf537a74f5294155ad234d380368d78c]) successfully deleted
I20260924 20:20:18.570462 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23956
W20260924 20:20:18.570478 23752 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:34635 (error 108)
I20260924 20:20:18.572243 24228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.571937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34635 (local address 127.0.0.1:60316)
0924 20:20:18.572110 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.572173 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:20:18.572417 23752 catalog_manager.cc:4584] TS c1fbcacef970490b9017e917afc2c41f (127.18.172.65:34635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce8fcad8ff404834985cb9caabfacf25: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34635: connect: Connection refused (error 111)
I20260924 20:20:18.576841 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24087
I20260924 20:20:18.583410 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 23734
2026-09-24T20:20:18Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1087 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 20:20:18.604652 19121 mini_oidc.cc:152] Starting JWKS server
I20260924 20:20:18.606204 19121 webserver.cc:460] Webserver started at https://127.0.0.1:42293/ using document root <none> and password file <none>
I20260924 20:20:18.606375 19121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:20:18.606652 19121 webserver.cc:460] Webserver started at http://127.0.0.1:34745/ using document root <none> and password file <none>
2026-09-24T20:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:18Z Disabled control of system clock
I20260924 20:20:18.615792 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34823
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:46659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34823
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42293/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/testchainca.cert with env {}
W20260924 20:20:18.726516 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:18.726786 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:18.726826 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:18.730974 24265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:18.731053 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:18.731069 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:18.731086 24265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:18.731107 24265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:20:18.731124 24265 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:20:18.731137 24265 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42293/jwks.json
I20260924 20:20:18.736418 24265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34823
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42293/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:18.737455 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:18.738422 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:18.747049 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:18.747169 24265 server_base.cc:1034] running on GCE node
W20260924 20:20:18.747058 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:18.747107 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:18.747891 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:18.748934 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:18.750125 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790281218750092 us; error 46 us; skew 500 ppm
I20260924 20:20:18.752247 24265 webserver.cc:460] Webserver started at http://127.18.172.126:34755/ using document root <none> and password file <none>
I20260924 20:20:18.752875 24265 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:18.752970 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:18.753201 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:18.755306 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "0327af608aea4baaa107625aabcf2f3f"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "f6408cac929f337789a8453a60b676e5"
server_key_iv: "218336ffa58b568307e7b12d6b773766"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.755905 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "0327af608aea4baaa107625aabcf2f3f"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "f6408cac929f337789a8453a60b676e5"
server_key_iv: "218336ffa58b568307e7b12d6b773766"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.759447 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:20:18.761838 24279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:18.763147 24265 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:18.763279 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "0327af608aea4baaa107625aabcf2f3f"
format_stamp: "Formatted at 2026-09-24 20:20:18 on dist-test-slave-m1qq"
server_key: "f6408cac929f337789a8453a60b676e5"
server_key_iv: "218336ffa58b568307e7b12d6b773766"
server_key_version: "encryptionkey@0"
I20260924 20:20:18.763389 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:18.776765 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:18.777506 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:18.777714 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:18.784773 24265 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:34823
I20260924 20:20:18.784780 24331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:34823 every 8 connection(s)
I20260924 20:20:18.785787 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:18.789054 24332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:18.791391 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24265
I20260924 20:20:18.791524 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:18.791810 19121 external_mini_cluster.cc:1428] Setting key dc6aa686b8b5195da3826f104a9c5ccf
I20260924 20:20:18.795485 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f: Bootstrap starting.
I20260924 20:20:18.797744 24332 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:18.798494 24332 log.cc:824] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:18.800671 24332 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f: No bootstrap required, opened a new log
I20260924 20:20:18.801309 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.793223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48710 (local address 127.18.172.126:34823)
0924 20:20:18.793880 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.793898 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.793955 (+ 57us) server_negotiation.cc:407] Connection header received
0924 20:20:18.794632 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.794651 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.795140 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.795497 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:18.795953 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.796948 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.797873 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.798354 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.798540 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:18.799449 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:18.799456 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:18.799473 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:18.799865 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:18.799920 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:18.799950 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:18.800026 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:18.800216 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 20:20:18.800510 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":123,"threads_started":1}
I20260924 20:20:18.804250 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:20:18.804420 24332 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } }
I20260924 20:20:18.804692 24332 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:18.804752 24332 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0327af608aea4baaa107625aabcf2f3f, State: Initialized, Role: FOLLOWER
I20260924 20:20:18.805207 24332 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } }
I20260924 20:20:18.805335 24332 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:18.805421 24332 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:18.805549 24332 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:18.806435 24332 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } }
I20260924 20:20:18.806788 24332 leader_election.cc:302] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0327af608aea4baaa107625aabcf2f3f; no voters:
I20260924 20:20:18.807159 24332 leader_election.cc:288] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:18.807276 24337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:18.807565 24337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [term 1 LEADER]: Becoming Leader. State: Replica: 0327af608aea4baaa107625aabcf2f3f, State: Running, Role: LEADER
I20260924 20:20:18.807876 24337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } }
I20260924 20:20:18.808391 24332 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:18.809556 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0327af608aea4baaa107625aabcf2f3f. Latest consensus state: current_term: 1 leader_uuid: "0327af608aea4baaa107625aabcf2f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } } }
I20260924 20:20:18.809687 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:18.809957 24338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0327af608aea4baaa107625aabcf2f3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0327af608aea4baaa107625aabcf2f3f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34823 } } }
I20260924 20:20:18.810110 24338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:18.810031 24346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:18.813768 24346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:18.819222 24346 catalog_manager.cc:1347] Generated new cluster ID: 21ff2c232e8945c4b1c3f5c46bb6e45e
I20260924 20:20:18.819303 24346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:18.856591 24346 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:18.857581 24346 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:18.865079 24346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0327af608aea4baaa107625aabcf2f3f: Generated new TSK 0
I20260924 20:20:18.865821 24346 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:20:18.903447 24362 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 20:20:18.903818 24362 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 20:20:18.905828 24361 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:18.886254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34823 (local address 127.0.0.1:48722)
0924 20:20:18.886591 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:18.886613 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:18.886733 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:18.887119 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:18.887121 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:18.887127 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:20:18.887224 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:18.887230 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.888300 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.888303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:18.888975 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:18.888985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.889650 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.889653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:18.889730 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:18.889777 (+ 47us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:20:18.904118 (+ 14341us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:20:18.904369 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.172.126:34823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":125,"threads_started":1}
I20260924 20:20:18.906423 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:18.886374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48722 (local address 127.18.172.126:34823)
0924 20:20:18.886519 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:18.886524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:18.886692 (+ 168us) server_negotiation.cc:407] Connection header received
0924 20:20:18.886898 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:18.886901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:18.886972 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:18.887115 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:20:18.887331 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.888163 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.889091 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:18.889442 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:18.889547 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:18.890001 (+ 454us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:20:18.903777 (+ 13776us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 20:20:18.906293 (+ 2516us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260924 20:20:18.906513 24335 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48722: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.18.172.126:34823: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42293/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 20:20:18.906754 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24265
2026-09-24T20:20:18Z chronyd exiting
I20260924 20:20:18.944619 19121 test_util.cc:175] -----------------------------------------------
I20260924 20:20:18.944708 19121 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790281188136637-19121-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (341 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 20:20:18.945698 19121 mini_oidc.cc:152] Starting JWKS server
I20260924 20:20:18.946552 19121 webserver.cc:460] Webserver started at https://127.0.0.1:41849/ using document root <none> and password file <none>
I20260924 20:20:18.946636 19121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:20:18.946861 19121 webserver.cc:460] Webserver started at http://127.0.0.1:46127/ using document root <none> and password file <none>
2026-09-24T20:20:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:18Z Disabled control of system clock
I20260924 20:20:18.955641 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43459
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:36871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43459
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41849/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/testchainca.cert with env {}
W20260924 20:20:19.065023 24368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:19.065295 24368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:19.065335 24368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:19.069269 24368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:19.069332 24368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:19.069348 24368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:19.069365 24368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:19.069380 24368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:20:19.069396 24368 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:20:19.069409 24368 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41849/jwks.json
I20260924 20:20:19.074694 24368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:36871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41849/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:19.075838 24368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:19.076817 24368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:19.085866 24374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.085862 24376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.085862 24373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:19.086212 24368 server_base.cc:1034] running on GCE node
I20260924 20:20:19.086769 24368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:19.087980 24368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:19.089172 24368 hybrid_clock.cc:648] HybridClock initialized: now 1790281219089153 us; error 34 us; skew 500 ppm
I20260924 20:20:19.091038 24368 webserver.cc:460] Webserver started at http://127.18.172.126:46175/ using document root <none> and password file <none>
I20260924 20:20:19.091645 24368 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:19.091738 24368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:19.091966 24368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:19.094344 24368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "742625bfdb444363bed636200b771f6a"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "8c7876ed9c3a72735830a7bfd32e7922"
server_key_iv: "5d3ef16e84c0736de6089a19d287118d"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.095052 24368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "742625bfdb444363bed636200b771f6a"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "8c7876ed9c3a72735830a7bfd32e7922"
server_key_iv: "5d3ef16e84c0736de6089a19d287118d"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.098982 24368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:20:19.101543 24382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:19.102984 24368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 20:20:19.103195 24368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "742625bfdb444363bed636200b771f6a"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "8c7876ed9c3a72735830a7bfd32e7922"
server_key_iv: "5d3ef16e84c0736de6089a19d287118d"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.103317 24368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:19.123296 24368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:19.123999 24368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:19.124195 24368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:19.131302 24368 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43459
I20260924 20:20:19.131351 24434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43459 every 8 connection(s)
I20260924 20:20:19.132432 24368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:19.135656 24435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:19.141264 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24368
I20260924 20:20:19.141394 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:19.141443 24435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a: Bootstrap starting.
I20260924 20:20:19.141706 19121 external_mini_cluster.cc:1428] Setting key a6525cc7b6105859721a8d95f9045308
I20260924 20:20:19.144162 24435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:19.144922 24435 log.cc:824] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:19.146901 24435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a: No bootstrap required, opened a new log
I20260924 20:20:19.150653 24435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } }
I20260924 20:20:19.150919 24435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:19.150995 24435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742625bfdb444363bed636200b771f6a, State: Initialized, Role: FOLLOWER
I20260924 20:20:19.151427 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.143210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.18.172.126:43459)
0924 20:20:19.143861 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.143875 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.143910 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:19.144619 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.144637 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.145090 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.145439 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:19.145871 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.146947 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.147885 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.148446 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.148627 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.149487 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:19.149492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:19.149507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:19.149888 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:19.149939 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:19.149958 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:19.150069 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:19.150215 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:20:19.150478 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":138,"threads_started":1}
I20260924 20:20:19.151441 24435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } }
I20260924 20:20:19.151655 24435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:19.151716 24435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:19.151801 24435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:19.152837 24435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } }
I20260924 20:20:19.153232 24435 leader_election.cc:302] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 742625bfdb444363bed636200b771f6a; no voters:
I20260924 20:20:19.153616 24435 leader_election.cc:288] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:19.153683 24440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:19.153887 24440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [term 1 LEADER]: Becoming Leader. State: Replica: 742625bfdb444363bed636200b771f6a, State: Running, Role: LEADER
I20260924 20:20:19.154175 24440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } }
I20260924 20:20:19.154495 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:20:19.154910 24435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:19.155381 24441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 742625bfdb444363bed636200b771f6a. Latest consensus state: current_term: 1 leader_uuid: "742625bfdb444363bed636200b771f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } } }
I20260924 20:20:19.155517 24441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.156060 24444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:19.156486 24442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "742625bfdb444363bed636200b771f6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "742625bfdb444363bed636200b771f6a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43459 } } }
I20260924 20:20:19.156726 24442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.159302 24444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:19.165416 24444 catalog_manager.cc:1347] Generated new cluster ID: 34c9c84704124ffdb6351387a7904810
I20260924 20:20:19.165516 24444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:19.194150 24444 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:19.195233 24444 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:19.202623 24444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 742625bfdb444363bed636200b771f6a: Generated new TSK 0
I20260924 20:20:19.203387 24444 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 20:20:19.230957 24465 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 20:20:19.231086 24465 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 20:20:19.231768 24438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.215330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38648 (local address 127.18.172.126:43459)
0924 20:20:19.215470 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.215476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.215654 (+ 178us) server_negotiation.cc:407] Connection header received
0924 20:20:19.215800 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.215802 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.215849 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.215939 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:20:19.216212 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.216802 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.218063 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.218278 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.218354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.218703 (+ 349us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:20:19.231407 (+ 12704us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:20:19.231648 (+ 241us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 20:20:19.231856 24438 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38648: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 20:20:19.232000 24464 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:19.215258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43459 (local address 127.0.0.1:38648)
0924 20:20:19.215554 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:19.215570 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:19.215688 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:19.215968 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:19.215971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:19.215980 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:20:19.216102 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.216109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.216917 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.216921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.217949 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.217960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.218408 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.218413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.218506 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.218536 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:20:19.231614 (+ 13078us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 20:20:19.231818 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.172.126:43459: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41849/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":97,"threads_started":1}
I20260924 20:20:19.232885 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24368
2026-09-24T20:20:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (326 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 20:20:19.272609 19121 mini_oidc.cc:152] Starting JWKS server
I20260924 20:20:19.273461 19121 webserver.cc:460] Webserver started at https://127.0.0.1:34073/ using document root <none> and password file <none>
I20260924 20:20:19.273543 19121 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 20:20:19.273808 19121 webserver.cc:460] Webserver started at http://127.0.0.1:41893/ using document root <none> and password file <none>
2026-09-24T20:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:19Z Disabled control of system clock
I20260924 20:20:19.282759 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43865
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:43337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43865
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34073/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:20:19.394562 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:19.394862 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:19.394903 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:19.399098 24471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:19.399158 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:19.399173 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:19.399191 24471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:19.399207 24471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:20:19.399223 24471 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:20:19.399236 24471 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34073/jwks.json
I20260924 20:20:19.403970 24471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:43337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43865
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34073/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:19.405000 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:19.406322 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:19.414959 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.415032 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.414961 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:19.415072 24471 server_base.cc:1034] running on GCE node
I20260924 20:20:19.415840 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:19.416910 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:19.418087 24471 hybrid_clock.cc:648] HybridClock initialized: now 1790281219418067 us; error 25 us; skew 500 ppm
I20260924 20:20:19.420100 24471 webserver.cc:460] Webserver started at http://127.18.172.126:34107/ using document root <none> and password file <none>
I20260924 20:20:19.420677 24471 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:19.420737 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:19.421008 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:19.423112 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "66c7bc85cb654bdabc1205bcd7baefa9"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "88a01d4aafc995fd0d4f6cf150c2ae15"
server_key_iv: "54e4f3e67b8e621ee4bcf0e78a350f5c"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.423672 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "66c7bc85cb654bdabc1205bcd7baefa9"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "88a01d4aafc995fd0d4f6cf150c2ae15"
server_key_iv: "54e4f3e67b8e621ee4bcf0e78a350f5c"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.427450 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 20:20:19.429960 24485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:19.431334 24471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:19.431483 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "66c7bc85cb654bdabc1205bcd7baefa9"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "88a01d4aafc995fd0d4f6cf150c2ae15"
server_key_iv: "54e4f3e67b8e621ee4bcf0e78a350f5c"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.431598 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:19.444809 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:19.445524 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:19.445735 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:19.452834 24471 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43865
I20260924 20:20:19.452847 24537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43865 every 8 connection(s)
I20260924 20:20:19.453785 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:19.457118 24538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:19.458642 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24471
I20260924 20:20:19.458750 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:19.459092 19121 external_mini_cluster.cc:1428] Setting key a28a376085e3bfd7276546db7ae8843f
I20260924 20:20:19.463726 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9: Bootstrap starting.
I20260924 20:20:19.466105 24538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:19.466988 24538 log.cc:824] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:19.468618 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.460533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.18.172.126:43865)
0924 20:20:19.461171 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.461185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.461219 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:20:19.461844 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.461860 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.462248 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.462638 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:19.463122 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.464154 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.465072 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.465540 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.465708 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.466627 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:19.466632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:19.466644 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:19.467078 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:19.467134 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:19.467172 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:19.467271 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:19.467450 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 20:20:19.467782 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":136,"threads_started":1}
I20260924 20:20:19.469355 24538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9: No bootstrap required, opened a new log
I20260924 20:20:19.471396 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:20:19.473343 24538 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } }
I20260924 20:20:19.473601 24538 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:19.473686 24538 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c7bc85cb654bdabc1205bcd7baefa9, State: Initialized, Role: FOLLOWER
I20260924 20:20:19.474125 24538 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } }
I20260924 20:20:19.474291 24538 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:19.474377 24538 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:19.474472 24538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:19.475476 24538 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } }
I20260924 20:20:19.475814 24538 leader_election.cc:302] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c7bc85cb654bdabc1205bcd7baefa9; no voters:
I20260924 20:20:19.476115 24538 leader_election.cc:288] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:19.476248 24543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:19.476483 24543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [term 1 LEADER]: Becoming Leader. State: Replica: 66c7bc85cb654bdabc1205bcd7baefa9, State: Running, Role: LEADER
I20260924 20:20:19.476786 24543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } }
I20260924 20:20:19.477432 24538 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:19.478363 24545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c7bc85cb654bdabc1205bcd7baefa9. Latest consensus state: current_term: 1 leader_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } } }
I20260924 20:20:19.478502 24545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.478860 24552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:19.478826 24544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c7bc85cb654bdabc1205bcd7baefa9" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43865 } } }
I20260924 20:20:19.479775 24544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.482028 24552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:19.488155 24552 catalog_manager.cc:1347] Generated new cluster ID: 1d525e138a6e4312b04de54fc67579d6
I20260924 20:20:19.488255 24552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:19.506640 24552 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:19.507946 24552 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:19.518446 24552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c7bc85cb654bdabc1205bcd7baefa9: Generated new TSK 0
I20260924 20:20:19.519477 24552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:19.541595 24541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.535461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37838 (local address 127.18.172.126:43865)
0924 20:20:19.535610 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.535616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.535847 (+ 231us) server_negotiation.cc:407] Connection header received
0924 20:20:19.536034 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.536036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.536086 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.536234 (+ 148us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:20:19.536449 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.536984 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.538290 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.538518 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.538609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.539053 (+ 444us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:20:19.541198 (+ 2145us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:20:19.541440 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 20:20:19.541668 24541 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37838: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 20:20:19.543525 24567 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:19.535368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43865 (local address 127.0.0.1:37838)
0924 20:20:19.535739 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:19.535754 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:19.535901 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:19.536236 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:19.536239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:19.536247 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:20:19.536362 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.536368 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.537103 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.537107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.538152 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.538163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.538660 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.538664 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.538755 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.538785 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:20:19.541375 (+ 2590us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 20:20:19.543345 (+ 1970us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.172.126:43865: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34073/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":281,"mutex_wait_us":1746,"thread_start_us":166,"threads_started":1}
I20260924 20:20:19.544144 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24471
E20260924 20:20:19.554636 24568 webserver.cc:533] Webserver: sslize failed:
E20260924 20:20:19.554739 24568 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T20:20:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (311 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T20:20:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:19Z Disabled control of system clock
I20260924 20:20:19.592707 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41751
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41751
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/testchainca.cert with env {}
W20260924 20:20:19.703459 24572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:19.703739 24572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:19.703780 24572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:19.707881 24572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:19.707949 24572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:19.707964 24572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:19.707983 24572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:19.707998 24572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 20:20:19.708014 24572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 20:20:19.708029 24572 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 20:20:19.713541 24572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41751
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:19.714735 24572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:19.715796 24572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:19.724526 24572 server_base.cc:1034] running on GCE node
W20260924 20:20:19.724536 24578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.724694 24580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:19.724535 24577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:19.725334 24572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:19.726485 24572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:19.727674 24572 hybrid_clock.cc:648] HybridClock initialized: now 1790281219727639 us; error 46 us; skew 500 ppm
I20260924 20:20:19.729526 24572 webserver.cc:460] Webserver started at http://127.18.172.126:34925/ using document root <none> and password file <none>
I20260924 20:20:19.730150 24572 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:19.730243 24572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:19.730482 24572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:19.732708 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "e31dae00ac254464bf45d15fb173a4cf"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "015f5240776c986853a5df494d733cbb"
server_key_iv: "162a3ae81cffaacf4af95fdcdb3a75d3"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.733289 24572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "e31dae00ac254464bf45d15fb173a4cf"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "015f5240776c986853a5df494d733cbb"
server_key_iv: "162a3ae81cffaacf4af95fdcdb3a75d3"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.737164 24572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:20:19.739879 24586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:19.741171 24572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:19.741322 24572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "e31dae00ac254464bf45d15fb173a4cf"
format_stamp: "Formatted at 2026-09-24 20:20:19 on dist-test-slave-m1qq"
server_key: "015f5240776c986853a5df494d733cbb"
server_key_iv: "162a3ae81cffaacf4af95fdcdb3a75d3"
server_key_version: "encryptionkey@0"
I20260924 20:20:19.741449 24572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:19.752789 24572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:19.753448 24572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:19.753662 24572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:19.760713 24572 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41751
I20260924 20:20:19.760723 24638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41751 every 8 connection(s)
I20260924 20:20:19.761632 24572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:19.764401 24639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:19.768213 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24572
I20260924 20:20:19.768345 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:19.768641 19121 external_mini_cluster.cc:1428] Setting key 2b75786a5d46b242798ff56367591691
I20260924 20:20:19.770427 24639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf: Bootstrap starting.
I20260924 20:20:19.772670 24639 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:19.773386 24639 log.cc:824] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:19.775498 24639 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf: No bootstrap required, opened a new log
I20260924 20:20:19.778107 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.770005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.18.172.126:41751)
0924 20:20:19.770694 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.770711 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.770746 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:19.771430 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.771448 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.771838 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.772180 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:19.772611 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.773672 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.774935 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.775485 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.775654 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.776430 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:19.776435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:19.776446 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:19.776820 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:19.776872 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:19.776891 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:19.776968 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:19.777137 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 20:20:19.777427 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":129,"threads_started":1}
I20260924 20:20:19.779381 24639 raft_consensus.cc:348] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } }
I20260924 20:20:19.779590 24639 raft_consensus.cc:374] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:19.779624 24639 raft_consensus.cc:729] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31dae00ac254464bf45d15fb173a4cf, State: Initialized, Role: FOLLOWER
I20260924 20:20:19.780114 24639 consensus_queue.cc:260] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } }
I20260924 20:20:19.780278 24639 raft_consensus.cc:388] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:19.780342 24639 raft_consensus.cc:482] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:19.780458 24639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:19.781028 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:20:19.781505 24639 raft_consensus.cc:504] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } }
I20260924 20:20:19.781876 24639 leader_election.cc:302] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e31dae00ac254464bf45d15fb173a4cf; no voters:
I20260924 20:20:19.782245 24639 leader_election.cc:288] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:19.782363 24644 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:19.782718 24644 raft_consensus.cc:686] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [term 1 LEADER]: Becoming Leader. State: Replica: e31dae00ac254464bf45d15fb173a4cf, State: Running, Role: LEADER
I20260924 20:20:19.783061 24644 consensus_queue.cc:237] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } }
I20260924 20:20:19.783547 24639 sys_catalog.cc:564] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:19.784792 24645 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e31dae00ac254464bf45d15fb173a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } } }
I20260924 20:20:19.784919 24645 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.785310 24653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:19.785287 24646 sys_catalog.cc:455] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e31dae00ac254464bf45d15fb173a4cf. Latest consensus state: current_term: 1 leader_uuid: "e31dae00ac254464bf45d15fb173a4cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e31dae00ac254464bf45d15fb173a4cf" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41751 } } }
I20260924 20:20:19.785435 24646 sys_catalog.cc:458] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:19.788256 24653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:19.793071 24653 catalog_manager.cc:1347] Generated new cluster ID: 8cae056b137245f1b6634f49be4e8417
I20260924 20:20:19.793143 24653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:19.803448 24653 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:19.804662 24653 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:19.810631 24653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e31dae00ac254464bf45d15fb173a4cf: Generated new TSK 0
I20260924 20:20:19.811440 24653 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:20.270601 24642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:19.819452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.18.172.126:41751)
0924 20:20:19.819589 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:19.819594 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:19.819830 (+ 236us) server_negotiation.cc:407] Connection header received
0924 20:20:19.819958 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:19.819961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:19.820017 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:19.820131 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0924 20:20:19.820325 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.821219 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.822131 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.822476 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.822588 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.822862 (+ 274us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 20:20:20.270179 (+447317us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:20:20.270475 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 20:20:20.270681 24642 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39622: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 20:20:20.270897 24668 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:19.819365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41751 (local address 127.0.0.1:39622)
0924 20:20:19.819748 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:19.819760 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:19.819853 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:19.820132 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:19.820134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:19.820140 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0924 20:20:19.820230 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.820234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.821363 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.821365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.822018 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:19.822027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:19.822605 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:19.822608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:19.822676 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:19.822697 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 20:20:20.270446 (+447749us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 20:20:20.270634 (+ 188us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.18.172.126:41751: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":103,"threads_started":1}
I20260924 20:20:20.271896 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24572
2026-09-24T20:20:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (707 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (125 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (124 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:20 dist-test-slave-m1qq krb5kdc[24680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:20 dist-test-slave-m1qq krb5kdc[24680](info): set up 2 sockets
Sep 24 20:20:20 dist-test-slave-m1qq krb5kdc[24680](info): commencing operation
krb5kdc: starting...
W20260924 20:20:21.608479 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:21 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:21.735783 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39495
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:21.851166 24693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:21.851459 24693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:21.851540 24693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:21.855657 24693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:21.855721 24693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:21.855741 24693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:21.855757 24693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:21.855772 24693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:21.861342 24693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:21.862457 24693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:21.863543 24693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:21.872156 24701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:21.872198 24693 server_base.cc:1034] running on GCE node
W20260924 20:20:21.872160 24699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:21.872156 24698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:21.872987 24693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:21.874174 24693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:21.875360 24693 hybrid_clock.cc:648] HybridClock initialized: now 1790281221875352 us; error 29 us; skew 500 ppm
Sep 24 20:20:21 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281221, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:21.882263 24693 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:21.883263 24693 webserver.cc:460] Webserver started at http://127.18.172.126:33243/ using document root <none> and password file <none>
I20260924 20:20:21.883838 24693 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:21.883929 24693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:21.884168 24693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:21.886260 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "ee7c124d1eb749a28106d1509aa9cc7c"
format_stamp: "Formatted at 2026-09-24 20:20:21 on dist-test-slave-m1qq"
server_key: "87bd10279eaa94981cc69a681545dbc6"
server_key_iv: "9ed4eb4f5a21e453aa3641d50cb425cf"
server_key_version: "encryptionkey@0"
I20260924 20:20:21.886842 24693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "ee7c124d1eb749a28106d1509aa9cc7c"
format_stamp: "Formatted at 2026-09-24 20:20:21 on dist-test-slave-m1qq"
server_key: "87bd10279eaa94981cc69a681545dbc6"
server_key_iv: "9ed4eb4f5a21e453aa3641d50cb425cf"
server_key_version: "encryptionkey@0"
I20260924 20:20:21.890511 24693 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:20:21.893035 24708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:21.894289 24693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:20:21.894532 24693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "ee7c124d1eb749a28106d1509aa9cc7c"
format_stamp: "Formatted at 2026-09-24 20:20:21 on dist-test-slave-m1qq"
server_key: "87bd10279eaa94981cc69a681545dbc6"
server_key_iv: "9ed4eb4f5a21e453aa3641d50cb425cf"
server_key_version: "encryptionkey@0"
I20260924 20:20:21.894646 24693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:21.917197 24693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:21.920663 24693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:21.920871 24693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:21.928160 24693 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:39495
I20260924 20:20:21.928179 24760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:39495 every 8 connection(s)
I20260924 20:20:21.929121 24693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:21.932022 24761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:21.933068 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24693
I20260924 20:20:21.933192 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:21.933514 19121 external_mini_cluster.cc:1428] Setting key ad973a0db480beb236ecb0423f6ff1ec
I20260924 20:20:21.938287 24761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: Bootstrap starting.
I20260924 20:20:21.940529 24761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:21.941233 24761 log.cc:824] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:20:21 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:21.943701 24761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: No bootstrap required, opened a new log
I20260924 20:20:21.947685 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:21.935025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52776 (local address 127.18.172.126:39495)
0924 20:20:21.935467 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:21.935483 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:21.935519 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:20:21.936149 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:21.936165 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:21.936491 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:21.936822 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:21.937664 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:21.938683 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:21.939586 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:21.940054 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:21.940226 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:21.943131 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:21.943136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:21.943148 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:21.943172 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:21.944878 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:21.945369 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:21.945373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:21.945377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:21.945481 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:21.945787 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:21.945789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:21.945790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:21.946175 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:21.946330 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:21.946601 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 20:20:21.946894 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":133,"threads_started":1}
I20260924 20:20:21.947949 24761 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } }
I20260924 20:20:21.948185 24761 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:21.948262 24761 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7c124d1eb749a28106d1509aa9cc7c, State: Initialized, Role: FOLLOWER
I20260924 20:20:21.948764 24761 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } }
I20260924 20:20:21.948927 24761 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:21.948992 24761 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:21.949084 24761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:21.950122 24761 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } }
I20260924 20:20:21.950414 24761 leader_election.cc:302] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee7c124d1eb749a28106d1509aa9cc7c; no voters:
I20260924 20:20:21.950670 24761 leader_election.cc:288] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:21.950790 24766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:21.951049 24766 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [term 1 LEADER]: Becoming Leader. State: Replica: ee7c124d1eb749a28106d1509aa9cc7c, State: Running, Role: LEADER
I20260924 20:20:21.951428 24766 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } }
I20260924 20:20:21.951910 24761 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:21.953738 24768 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee7c124d1eb749a28106d1509aa9cc7c. Latest consensus state: current_term: 1 leader_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } } }
I20260924 20:20:21.954037 24768 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:21.953948 24767 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee7c124d1eb749a28106d1509aa9cc7c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39495 } } }
I20260924 20:20:21.954530 24767 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c [sys.catalog]: This master's current role is: LEADER
W20260924 20:20:21.956548 24780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:20:21.956615 24780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:20:21.956699 24782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:21.957403 24782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:21.962888 24782 catalog_manager.cc:1347] Generated new cluster ID: 906cb42240fd4b12ad88c8927b50ae93
I20260924 20:20:21.962952 24782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:21.992609 24782 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:21.993908 24782 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:22.005261 24782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee7c124d1eb749a28106d1509aa9cc7c: Generated new TSK 0
I20260924 20:20:22.006160 24782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:22.035802 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:22.147737 24787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:22.148018 24787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:22.148108 24787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:22.152367 24787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:22.152432 24787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:22.152551 24787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:22.158087 24787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:22.159210 24787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:22.160241 24787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:22.170245 24787 server_base.cc:1034] running on GCE node
W20260924 20:20:22.170186 24793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:22.170186 24792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:22.170202 24795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:22.170830 24787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:22.171547 24787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:22.172734 24787 hybrid_clock.cc:648] HybridClock initialized: now 1790281222172697 us; error 46 us; skew 500 ppm
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.175621 24787 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:22.176512 24787 webserver.cc:460] Webserver started at http://127.18.172.65:45769/ using document root <none> and password file <none>
I20260924 20:20:22.177119 24787 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:22.177207 24787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:22.177450 24787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:22.179522 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "1fdac8bd25394d1883694be7132dc58c"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "7a3ed6d6145a6a81515399c3afc32ca9"
server_key_iv: "8137d25a6d950f42bc41eda521ef6142"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.180070 24787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "1fdac8bd25394d1883694be7132dc58c"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "7a3ed6d6145a6a81515399c3afc32ca9"
server_key_iv: "8137d25a6d950f42bc41eda521ef6142"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.183801 24787 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:20:22.186210 24802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.187531 24787 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:20:22.187661 24787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "1fdac8bd25394d1883694be7132dc58c"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "7a3ed6d6145a6a81515399c3afc32ca9"
server_key_iv: "8137d25a6d950f42bc41eda521ef6142"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.187773 24787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:22.202150 24787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:22.205538 24787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:22.205753 24787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:22.206358 24787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:22.207242 24787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:22.207316 24787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.207412 24787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:22.207464 24787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.216034 24787 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:39303
I20260924 20:20:22.216053 24915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:39303 every 8 connection(s)
I20260924 20:20:22.216903 24787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:22.222105 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24787
I20260924 20:20:22.222245 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:22.222532 19121 external_mini_cluster.cc:1428] Setting key 5014fcfc3e7040ab7b79b3e985e90683
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:22.231547 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.218686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:39267 (local address 127.18.172.126:39495)
0924 20:20:22.218823 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.218827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.219780 (+ 953us) server_negotiation.cc:407] Connection header received
0924 20:20:22.221307 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.221311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.221371 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.221476 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.222783 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.223465 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.224714 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.224949 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.225060 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.227816 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.227819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.227821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.227845 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.229356 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.229880 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.229885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.229886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.229969 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.230322 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.230327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.230329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.230516 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.230633 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.231212 (+ 579us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.231361 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:22.232396 24918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.219135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39495 (local address 127.18.172.65:39267)
0924 20:20:22.219623 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:22.219669 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:22.221087 (+ 1418us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:22.221650 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:22.221656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:22.222007 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:22.222633 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.222650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.223593 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.223596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.224588 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.224597 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.225110 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.225114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.225283 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.225906 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:22.225922 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:22.227632 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:22.229483 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.229489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.229507 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.229773 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.230070 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.230080 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.230082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.230227 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.230631 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:22.230638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:22.230895 (+ 257us) client_negotiation.cc:769] Sending connection context
0924 20:20:22.231162 (+ 267us) client_negotiation.cc:240] Negotiation successful
0924 20:20:22.231516 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":116,"threads_started":1}
I20260924 20:20:22.233713 24916 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39495
I20260924 20:20:22.233976 24916 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:22.234668 24916 heartbeater.cc:511] Master 127.18.172.126:39495 requested a full tablet report, sending...
I20260924 20:20:22.236311 24725 ts_manager.cc:194] Registered new tserver with Master: 1fdac8bd25394d1883694be7132dc58c (127.18.172.65:39303)
I20260924 20:20:22.237895 24725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:39267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:22.250523 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:22.369855 24921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:22.370271 24921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:22.370357 24921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:22.374691 24921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:22.374809 24921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:22.374913 24921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:22.380666 24921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:22.381980 24921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:22.383116 24921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:22.393175 24926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:22.393236 24921 server_base.cc:1034] running on GCE node
W20260924 20:20:22.393191 24927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:22.393191 24929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:22.393800 24921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:22.394397 24921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:22.395615 24921 hybrid_clock.cc:648] HybridClock initialized: now 1790281222395601 us; error 28 us; skew 500 ppm
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.398532 24921 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:22.399583 24921 webserver.cc:460] Webserver started at http://127.18.172.66:38351/ using document root <none> and password file <none>
I20260924 20:20:22.400458 24921 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:22.400532 24921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:22.400707 24921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:22.402957 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "353b353450bf4470be2eb835e6b2b16a"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "8cfe6f99e7a756144c09dbfb70c5ec76"
server_key_iv: "75c40036c5162a7ddb1a0b0debd4713f"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.403563 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "353b353450bf4470be2eb835e6b2b16a"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "8cfe6f99e7a756144c09dbfb70c5ec76"
server_key_iv: "75c40036c5162a7ddb1a0b0debd4713f"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.407222 24921 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:22.409708 24936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.410943 24921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:22.411136 24921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "353b353450bf4470be2eb835e6b2b16a"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "8cfe6f99e7a756144c09dbfb70c5ec76"
server_key_iv: "75c40036c5162a7ddb1a0b0debd4713f"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.411253 24921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:22.424746 24921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:22.428059 24921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:22.428267 24921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:22.428835 24921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:22.429625 24921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:22.429698 24921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.429770 24921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:22.429813 24921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.438561 24921 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:45415
I20260924 20:20:22.438588 25049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:45415 every 8 connection(s)
I20260924 20:20:22.439471 24921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:22.447345 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 24921
I20260924 20:20:22.447443 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:22.447978 19121 external_mini_cluster.cc:1428] Setting key a6d445b3cd8d7c3e6623f1d15aefc65c
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:22.454046 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.441309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:46983 (local address 127.18.172.126:39495)
0924 20:20:22.441479 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.441483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.442310 (+ 827us) server_negotiation.cc:407] Connection header received
0924 20:20:22.443664 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.443667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.443714 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.443797 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.444935 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.445453 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.446555 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.446776 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.446853 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.449706 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.449710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.449712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.449738 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.451408 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.452092 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.452095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.452096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.452211 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.452686 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.452693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.452697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.452976 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.453077 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.453658 (+ 581us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.453803 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:20:22.455114 25052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.441715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39495 (local address 127.18.172.66:46983)
0924 20:20:22.442166 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:22.442207 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:22.443452 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:22.443959 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:22.443966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:22.444327 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:22.444815 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.444828 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.445562 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.445564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.446451 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.446460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.446973 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.446978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.447256 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.447848 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:22.447864 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:22.449541 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:22.451533 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.451542 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.451561 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.451990 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.452369 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.452373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.452374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.452576 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.453094 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:22.453099 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:22.453386 (+ 287us) client_negotiation.cc:769] Sending connection context
0924 20:20:22.453584 (+ 198us) client_negotiation.cc:240] Negotiation successful
0924 20:20:22.453902 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260924 20:20:22.456339 25050 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39495
I20260924 20:20:22.456604 25050 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:22.457085 25050 heartbeater.cc:511] Master 127.18.172.126:39495 requested a full tablet report, sending...
I20260924 20:20:22.458155 24725 ts_manager.cc:194] Registered new tserver with Master: 353b353450bf4470be2eb835e6b2b16a (127.18.172.66:45415)
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:22.458933 24725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:46983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:22.477854 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:42771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:22.588294 25055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:22.588560 25055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:22.588603 25055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:22.592674 25055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:22.592739 25055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:22.592797 25055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:22.598148 25055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:39495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:22.599223 25055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:22.600219 25055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:22.609685 25061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:22.609685 25060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:22.609724 25063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:22.609939 25055 server_base.cc:1034] running on GCE node
I20260924 20:20:22.610491 25055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:22.611202 25055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:22.612398 25055 hybrid_clock.cc:648] HybridClock initialized: now 1790281222612373 us; error 35 us; skew 500 ppm
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.615564 25055 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:22.616660 25055 webserver.cc:460] Webserver started at http://127.18.172.67:40079/ using document root <none> and password file <none>
I20260924 20:20:22.617239 25055 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:22.617332 25055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:22.617590 25055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:22.619745 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "a860a858afbd47a7aba5c09d0b83fd8b"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "2237b308f501430991b2e9aa6c72ee7e"
server_key_iv: "b45e8aff044fc5887a83e29e3e88899d"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.620296 25055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "a860a858afbd47a7aba5c09d0b83fd8b"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "2237b308f501430991b2e9aa6c72ee7e"
server_key_iv: "b45e8aff044fc5887a83e29e3e88899d"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.624092 25055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:20:22.626507 25070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.627866 25055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:22.628005 25055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "a860a858afbd47a7aba5c09d0b83fd8b"
format_stamp: "Formatted at 2026-09-24 20:20:22 on dist-test-slave-m1qq"
server_key: "2237b308f501430991b2e9aa6c72ee7e"
server_key_iv: "b45e8aff044fc5887a83e29e3e88899d"
server_key_version: "encryptionkey@0"
I20260924 20:20:22.628118 25055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:22.638396 25055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:22.641171 25055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:22.641391 25055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:22.641970 25055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:22.642797 25055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:22.642874 25055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.642942 25055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:22.642992 25055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:22.652084 25055 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:43745
I20260924 20:20:22.652101 25183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:43745 every 8 connection(s)
I20260924 20:20:22.653131 25055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:22.653988 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25055
I20260924 20:20:22.654102 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:22.654376 19121 external_mini_cluster.cc:1428] Setting key 081d9922df2b6923bb98c3804658c454
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:22.666783 24764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.655152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:50415 (local address 127.18.172.126:39495)
0924 20:20:22.655279 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.655283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.656377 (+ 1094us) server_negotiation.cc:407] Connection header received
0924 20:20:22.657795 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.657798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.657843 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.657923 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.659061 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.659655 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.660779 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.661017 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.661103 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.663506 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.663509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.663511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.663535 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.664770 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.665214 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.665216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.665218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.665293 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.665723 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.665725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.665726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.665910 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.665986 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.666508 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.666617 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 20:20:22.667616 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.655637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39495 (local address 127.18.172.67:50415)
0924 20:20:22.656229 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:22.656287 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:22.657590 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:22.658051 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:22.658057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:22.658408 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:22.658888 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.658901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.659769 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.659772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.660668 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:22.660677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.661146 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.661148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:22.661287 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.661899 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:22.661915 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:22.663350 (+ 1435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:22.664871 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.664875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.664886 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.665122 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.665482 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:22.665484 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:22.665485 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:22.665639 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:22.665996 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:22.666000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:22.666217 (+ 217us) client_negotiation.cc:769] Sending connection context
0924 20:20:22.666372 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 20:20:22.666629 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":152,"threads_started":1}
I20260924 20:20:22.668723 25184 heartbeater.cc:347] Connected to a master server at 127.18.172.126:39495
I20260924 20:20:22.668943 25184 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:22.669488 25184 heartbeater.cc:511] Master 127.18.172.126:39495 requested a full tablet report, sending...
I20260924 20:20:22.670488 24725 ts_manager.cc:194] Registered new tserver with Master: a860a858afbd47a7aba5c09d0b83fd8b (127.18.172.67:43745)
I20260924 20:20:22.671124 24725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:50415
I20260924 20:20:22.680343 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/test-admin.keytab.
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.696835 19121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 20:20:22.700312 25191 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:22.698367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37986 (local address 127.18.172.65:39303)
0924 20:20:22.698711 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.698735 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.698762 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:20:22.698912 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.698918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.699240 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.699412 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.700076 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37986: BlockingRecv error: recv got EOF from 127.0.0.1:37986 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":123,"threads_started":1}
W20260924 20:20:22.700342 19121 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.701556 19121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.702586 19121 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:20:22.712747 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.703754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37988 (local address 127.18.172.65:39303)
0924 20:20:22.703892 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.703895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.703909 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:22.703969 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.703971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.704007 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.704078 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.704642 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.705430 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.706263 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.706655 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.706739 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.709055 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.709064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.709079 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.709104 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.710796 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.711331 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.711337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.711338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.711419 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.711697 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.711702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.711703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.711956 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.712206 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.712427 (+ 221us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.712606 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260924 20:20:22.716938 25191 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:22.715996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37996 (local address 127.18.172.65:39303)
0924 20:20:22.716130 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.716133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.716147 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:22.716272 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.716274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.716309 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.716375 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.716826 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37996: BlockingRecv error: recv got EOF from 127.0.0.1:37996 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260924 20:20:22.716992 19121 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.718242 19121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.719293 19121 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:20:22.729329 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.720221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.18.172.65:39303)
0924 20:20:22.720380 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.720385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.720559 (+ 174us) server_negotiation.cc:407] Connection header received
0924 20:20:22.720728 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.720732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.720785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.720879 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.721436 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.722100 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.723057 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.723305 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.723442 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.726023 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.726026 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.726029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.726054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.727528 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.728013 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.728025 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.728027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.728105 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.728398 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.728403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.728405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.728605 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.728757 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.729014 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.729142 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 20:20:22.732748 25191 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:22.731522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.18.172.65:39303)
0924 20:20:22.731675 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.731680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.731882 (+ 202us) server_negotiation.cc:407] Connection header received
0924 20:20:22.732022 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.732025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.732073 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.732147 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.732597 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38016: BlockingRecv error: recv got EOF from 127.0.0.1:38016 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:20:22.732903 19121 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790281188136637-19121-0/krb5kdc/krb5cc' of type FILE
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.734084 19121 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:22.735140 19121 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[24680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:20:22.745316 25191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:22.736241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38030 (local address 127.18.172.65:39303)
0924 20:20:22.736404 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:22.736410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:22.736632 (+ 222us) server_negotiation.cc:407] Connection header received
0924 20:20:22.736806 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:22.736810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:22.736867 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:22.736984 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:22.737572 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.738150 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.739166 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:22.739415 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:22.739535 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:22.742140 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:22.742144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:22.742154 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:22.742180 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:22.743566 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.744017 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.744021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.744022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.744104 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:22.744476 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:22.744478 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:22.744479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:22.744641 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:22.744783 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:22.745035 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 20:20:22.745143 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 20:20:22.746528 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24787
I20260924 20:20:22.752254 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24921
I20260924 20:20:22.757712 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25055
I20260924 20:20:22.763514 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 24693
2026-09-24T20:20:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2243 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[25204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[25204](info): set up 2 sockets
Sep 24 20:20:22 dist-test-slave-m1qq krb5kdc[25204](info): commencing operation
krb5kdc: starting...
W20260924 20:20:23.804533 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:23 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab.
Entry for principal oryx/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab.
I20260924 20:20:23.938643 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34959
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab
--principal=oryx/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:24.059434 25217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:24.059805 25217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:24.059885 25217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:24.067615 25217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:24.067714 25217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:24.067756 25217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:24.067850 25217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:24.067898 25217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:24.079376 25217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab
--principal=oryx/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:24.080973 25217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:24.082360 25217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:24.091185 25222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.091188 25225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.091199 25223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:24.091477 25217 server_base.cc:1034] running on GCE node
I20260924 20:20:24.092149 25217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:24.093240 25217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:24.094434 25217 hybrid_clock.cc:648] HybridClock initialized: now 1790281224094398 us; error 50 us; skew 500 ppm
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:24.097412 25217 init.cc:377] Logged in from keytab as oryx/127.18.172.126@KRBTEST.COM (short username oryx)
I20260924 20:20:24.098349 25217 webserver.cc:460] Webserver started at http://127.18.172.126:44801/ using document root <none> and password file <none>
I20260924 20:20:24.098903 25217 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:24.098961 25217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:24.099244 25217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:24.101245 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "6c03f18dff984dd893e3f7719b2526da"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cca9f26b93b0de9af599e1e5d2c7efe4"
server_key_iv: "40a409280d4c05beaa4666e9291b70eb"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.101752 25217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "6c03f18dff984dd893e3f7719b2526da"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cca9f26b93b0de9af599e1e5d2c7efe4"
server_key_iv: "40a409280d4c05beaa4666e9291b70eb"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.105587 25217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 20:20:24.108013 25232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.109301 25217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260924 20:20:24.109431 25217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "6c03f18dff984dd893e3f7719b2526da"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cca9f26b93b0de9af599e1e5d2c7efe4"
server_key_iv: "40a409280d4c05beaa4666e9291b70eb"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.109537 25217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:24.148682 25217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:24.152329 25217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:24.152650 25217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:24.160140 25217 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:34959
I20260924 20:20:24.160459 25284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:34959 every 8 connection(s)
I20260924 20:20:24.161226 25217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:24.164253 25285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:24.165715 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25217
I20260924 20:20:24.165831 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:24.166196 19121 external_mini_cluster.cc:1428] Setting key e683d841b99af4b0dfb3cbcff8edc5ce
I20260924 20:20:24.170733 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Bootstrap starting.
I20260924 20:20:24.173581 25285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:24.174356 25285 log.cc:824] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:24.176596 25285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: No bootstrap required, opened a new log
I20260924 20:20:24.180183 25285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } }
I20260924 20:20:24.180405 25285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:24.180464 25285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c03f18dff984dd893e3f7719b2526da, State: Initialized, Role: FOLLOWER
I20260924 20:20:24.180600 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.167765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.18.172.126:34959)
0924 20:20:24.168241 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:24.168257 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:24.168295 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:20:24.168960 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:24.168989 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:24.169335 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:24.169691 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:24.170494 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.171598 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.172466 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.172947 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.173127 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.175896 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:24.175901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:24.175915 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:24.175941 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:24.177719 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.178170 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.178175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.178180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.178294 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.178578 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.178583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.178585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.178972 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:24.179250 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:24.179481 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 20:20:24.179771 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":146,"threads_started":1}
I20260924 20:20:24.180971 25285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } }
I20260924 20:20:24.181128 25285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:24.181192 25285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:24.181285 25285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:24.182284 25285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } }
I20260924 20:20:24.182633 25285 leader_election.cc:302] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c03f18dff984dd893e3f7719b2526da; no voters:
I20260924 20:20:24.183036 25285 leader_election.cc:288] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:24.183182 25290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:24.183420 25290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 LEADER]: Becoming Leader. State: Replica: 6c03f18dff984dd893e3f7719b2526da, State: Running, Role: LEADER
I20260924 20:20:24.183719 25290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } }
I20260924 20:20:24.184278 25285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:24.185528 25292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c03f18dff984dd893e3f7719b2526da. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } }
I20260924 20:20:24.185493 25291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } }
I20260924 20:20:24.185665 25292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:24.185665 25291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:24.186107 25299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:24.189477 25299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:24.195875 25299 catalog_manager.cc:1347] Generated new cluster ID: ff1caf86e16841febce96b6acb2c6866
I20260924 20:20:24.195956 25299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:24.206992 25299 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:24.208472 25299 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:24.213797 25299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Generated new TSK 0
I20260924 20:20:24.214721 25299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
I20260924 20:20:24.251156 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:24.364634 25311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:24.364944 25311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:24.365041 25311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:24.369627 25311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:24.369746 25311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:24.369849 25311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:24.375564 25311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:24.376730 25311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:24.377833 25311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:24.387889 25316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.387884 25317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.387904 25319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:24.388228 25311 server_base.cc:1034] running on GCE node
I20260924 20:20:24.388597 25311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:24.389163 25311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:24.390321 25311 hybrid_clock.cc:648] HybridClock initialized: now 1790281224390308 us; error 23 us; skew 500 ppm
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:24.393018 25311 init.cc:377] Logged in from keytab as oryx/127.18.172.65@KRBTEST.COM (short username oryx)
I20260924 20:20:24.393823 25311 webserver.cc:460] Webserver started at http://127.18.172.65:41965/ using document root <none> and password file <none>
I20260924 20:20:24.394341 25311 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:24.394397 25311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:24.394559 25311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:24.396620 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "fb843e144828415e8953c52fb2213bb5"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "18fed9ce3ca0bccabfc3597acc58acb6"
server_key_iv: "f5a928e225539306ee8191809b52c582"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.397130 25311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "fb843e144828415e8953c52fb2213bb5"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "18fed9ce3ca0bccabfc3597acc58acb6"
server_key_iv: "f5a928e225539306ee8191809b52c582"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.400919 25311 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:20:24.403353 25326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.404587 25311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:20:24.404681 25311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fb843e144828415e8953c52fb2213bb5"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "18fed9ce3ca0bccabfc3597acc58acb6"
server_key_iv: "f5a928e225539306ee8191809b52c582"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.404819 25311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:24.419885 25311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:24.423166 25311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:24.423405 25311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:24.424053 25311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:24.424942 25311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:24.424993 25311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.425069 25311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:24.425104 25311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.435281 25311 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:43265
I20260924 20:20:24.435343 25439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:43265 every 8 connection(s)
I20260924 20:20:24.436228 25311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:24.438051 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25311
I20260924 20:20:24.438205 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:24.438550 19121 external_mini_cluster.cc:1428] Setting key 32d4f3e4168a96e095e97350e672869c
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:24.453578 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.438329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:44447 (local address 127.18.172.126:34959)
0924 20:20:24.438514 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:24.438520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:24.439823 (+ 1303us) server_negotiation.cc:407] Connection header received
0924 20:20:24.441347 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:24.441352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:24.441416 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:24.441525 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:24.443325 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.443879 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.445953 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.446182 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.446266 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.449712 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:24.449717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:24.449720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:24.449755 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:24.451181 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.451737 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.451740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.451741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.451822 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.452292 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.452294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.452295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.452472 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:24.452546 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:24.453215 (+ 669us) server_negotiation.cc:299] Negotiation successful
0924 20:20:24.453365 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:20:24.454679 25442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.438991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:44447)
0924 20:20:24.439615 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:24.439721 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:24.441141 (+ 1420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:24.441726 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:24.441739 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:24.442314 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:24.443140 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.443166 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.444021 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.444028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.445805 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.445821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.446361 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.446369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.446624 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.447403 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:24.447426 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:24.449503 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:24.451291 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.451297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.451310 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.451611 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.451927 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.451929 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.451931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.452072 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.452584 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:24.452590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:24.452838 (+ 248us) client_negotiation.cc:769] Sending connection context
0924 20:20:24.453092 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 20:20:24.453453 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":153,"threads_started":1}
I20260924 20:20:24.456267 25440 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:24.456578 25440 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:24.457108 25440 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:24.458771 25249 ts_manager.cc:194] Registered new tserver with Master: fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265)
I20260924 20:20:24.459990 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:44447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
I20260924 20:20:24.470924 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:24.584113 25445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:24.584419 25445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:24.584522 25445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:24.588802 25445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:24.588891 25445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:24.588984 25445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:24.594324 25445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:24.595495 25445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:24.596539 25445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:24.606143 25450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.606115 25451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.606117 25453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:24.606552 25445 server_base.cc:1034] running on GCE node
I20260924 20:20:24.606947 25445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:24.607625 25445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:24.608794 25445 hybrid_clock.cc:648] HybridClock initialized: now 1790281224608780 us; error 25 us; skew 500 ppm
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:24.611431 25445 init.cc:377] Logged in from keytab as oryx/127.18.172.66@KRBTEST.COM (short username oryx)
I20260924 20:20:24.612313 25445 webserver.cc:460] Webserver started at http://127.18.172.66:44111/ using document root <none> and password file <none>
I20260924 20:20:24.612872 25445 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:24.612962 25445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:24.613198 25445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:24.615392 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "d5d7e03495e748f28e6c45ab824b6c98"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cf138f885595711df3ae7b06f46bfd20"
server_key_iv: "b722b9481e6e488d0ff03e9a63aac052"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.616008 25445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "d5d7e03495e748f28e6c45ab824b6c98"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cf138f885595711df3ae7b06f46bfd20"
server_key_iv: "b722b9481e6e488d0ff03e9a63aac052"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.619681 25445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260924 20:20:24.622051 25460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.623399 25445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:24.623535 25445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "d5d7e03495e748f28e6c45ab824b6c98"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cf138f885595711df3ae7b06f46bfd20"
server_key_iv: "b722b9481e6e488d0ff03e9a63aac052"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.623649 25445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:24.639519 25445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:24.643669 25445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:24.643853 25445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:24.644395 25445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:24.645221 25445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:24.645274 25445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.645315 25445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:24.645330 25445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.653896 25445 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:41779
I20260924 20:20:24.653950 25573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:41779 every 8 connection(s)
I20260924 20:20:24.654855 25445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:24.657164 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25445
I20260924 20:20:24.657284 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:24.657658 19121 external_mini_cluster.cc:1428] Setting key e539a5a27fbf5b37d984512cde41d70a
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:24.670287 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.656851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:35895 (local address 127.18.172.126:34959)
0924 20:20:24.657012 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:24.657016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:24.658203 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 20:20:24.659849 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:24.659855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:24.659914 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:24.660010 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:24.661472 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.661978 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.663313 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.663533 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.663613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.666543 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:24.666546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:24.666548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:24.666571 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:24.668104 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.668676 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.668681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.668683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.668767 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.669134 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.669137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.669139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.669376 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:24.669483 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:24.669988 (+ 505us) server_negotiation.cc:299] Negotiation successful
0924 20:20:24.670114 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:20:24.671218 25576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.657398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:35895)
0924 20:20:24.657997 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:24.658076 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:24.659637 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:24.660171 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:24.660178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:24.660550 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:24.661307 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.661329 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.662107 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.662110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.663195 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.663210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.663692 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.663700 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.663957 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.664639 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:24.664667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:24.666365 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:24.668250 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.668256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.668271 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.668566 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.668882 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.668885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.668887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.669023 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.669481 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:24.669488 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:24.669730 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 20:20:24.669936 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 20:20:24.670254 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.67@KRBTEST.COM" created.
I20260924 20:20:24.672610 25574 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:24.672880 25574 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:24.673393 25574 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:24.674597 25249 ts_manager.cc:194] Registered new tserver with Master: d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779)
I20260924 20:20:24.675239 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:35895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
I20260924 20:20:24.690418 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:24.807804 25579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:24.808177 25579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:24.808274 25579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:24.812920 25579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:24.813045 25579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:24.813163 25579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:24.819334 25579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:24.820587 25579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:24.821749 25579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:24.832281 25585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.832310 25587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:24.832329 25584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:24.832685 25579 server_base.cc:1034] running on GCE node
I20260924 20:20:24.833134 25579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:24.833797 25579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:24.834954 25579 hybrid_clock.cc:648] HybridClock initialized: now 1790281224834939 us; error 28 us; skew 500 ppm
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:24.837893 25579 init.cc:377] Logged in from keytab as oryx/127.18.172.67@KRBTEST.COM (short username oryx)
I20260924 20:20:24.838824 25579 webserver.cc:460] Webserver started at http://127.18.172.67:46233/ using document root <none> and password file <none>
I20260924 20:20:24.839450 25579 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:24.839514 25579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:24.839740 25579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:24.841817 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "8edad0b4304c42998162c48e774d2a0d"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "4ba85f566a1c3c9de2dfb4852af692da"
server_key_iv: "8ea6807ab19f35f7134206206435a8ea"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.842399 25579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "8edad0b4304c42998162c48e774d2a0d"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "4ba85f566a1c3c9de2dfb4852af692da"
server_key_iv: "8ea6807ab19f35f7134206206435a8ea"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.846468 25579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260924 20:20:24.849062 25594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.850384 25579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:24.850519 25579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "8edad0b4304c42998162c48e774d2a0d"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "4ba85f566a1c3c9de2dfb4852af692da"
server_key_iv: "8ea6807ab19f35f7134206206435a8ea"
server_key_version: "encryptionkey@0"
I20260924 20:20:24.850637 25579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:24.872455 25579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:24.876883 25579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:24.877125 25579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:24.877872 25579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:24.878991 25579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:24.879079 25579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.879136 25579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:24.879172 25579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:24.888731 25579 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:35541
I20260924 20:20:24.888756 25707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:35541 every 8 connection(s)
I20260924 20:20:24.889631 25579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:24.897099 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25579
I20260924 20:20:24.897228 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:24.897557 19121 external_mini_cluster.cc:1428] Setting key 6182757c403616b7c8f59eaf00dcb8f0
Sep 24 20:20:24 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:24.903438 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.891482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:41237 (local address 127.18.172.126:34959)
0924 20:20:24.891623 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:24.891627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:24.892495 (+ 868us) server_negotiation.cc:407] Connection header received
0924 20:20:24.893813 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:24.893816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:24.893864 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:24.893940 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:24.895240 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.895766 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.896917 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.897126 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.897241 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.900206 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:24.900209 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:24.900212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:24.900236 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:24.901414 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.901916 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.901920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.901922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.902004 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:24.902338 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:24.902343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:24.902345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:24.902529 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:24.902636 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:24.903160 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 20:20:24.903276 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:24.904228 25710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:24.891887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:41237)
0924 20:20:24.892325 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:24.892383 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:24.893622 (+ 1239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:24.894132 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:24.894138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:24.894514 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:24.895073 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.895112 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.895880 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.895882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.896771 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:24.896780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:24.897302 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:24.897306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:24.897522 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:24.898233 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:24.898255 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:24.900050 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:24.901529 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.901534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.901548 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.901805 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.902099 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:24.902103 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:24.902105 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:24.902242 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:24.902647 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:24.902655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:24.902897 (+ 242us) client_negotiation.cc:769] Sending connection context
0924 20:20:24.903106 (+ 209us) client_negotiation.cc:240] Negotiation successful
0924 20:20:24.903420 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":110,"threads_started":1}
I20260924 20:20:24.905349 25708 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:24.905611 25708 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:24.906090 25708 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:24.907095 25249 ts_manager.cc:194] Registered new tserver with Master: 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:24.907661 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:41237
I20260924 20:20:24.911816 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.125.keytab.
Entry for principal oryx/127.18.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.125.keytab.
I20260924 20:20:24.938498 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:43773
--webserver_interface=127.18.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.125.keytab
--principal=oryx/127.18.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34959,127.18.172.125:43773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:25.050608 25713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:25.050889 25713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:25.050931 25713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:25.055250 25713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:25.055308 25713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:25.055328 25713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:25.055344 25713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:25.055359 25713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:25.060738 25713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34959,127.18.172.125:43773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.125.keytab
--principal=oryx/127.18.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:43773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
W20260924 20:20:25.060945 25713 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.18.172.126:34959,127.18.172.125:43773'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 20:20:25.061800 25713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:25.062753 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:25.071388 25718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.071519 25721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.071586 25719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.071704 25713 server_base.cc:1034] running on GCE node
I20260924 20:20:25.072353 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.073525 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.074723 25713 hybrid_clock.cc:648] HybridClock initialized: now 1790281225074682 us; error 48 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.077608 25713 init.cc:377] Logged in from keytab as oryx/127.18.172.125@KRBTEST.COM (short username oryx)
I20260924 20:20:25.078541 25713 webserver.cc:460] Webserver started at http://127.18.172.125:33101/ using document root <none> and password file <none>
I20260924 20:20:25.079180 25713 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.079272 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.079496 25713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:25.081522 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/instance:
uuid: "b7700bf492734b54b013d5d105ecddea"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "cd511beb992d87e30b04347c8159352e"
server_key_iv: "b6e98040f01194cccaf98487dc81be81"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.082149 25713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal/instance:
uuid: "b7700bf492734b54b013d5d105ecddea"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "cd511beb992d87e30b04347c8159352e"
server_key_iv: "b6e98040f01194cccaf98487dc81be81"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.086453 25713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:20:25.088989 25728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.090348 25713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:25.090530 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "b7700bf492734b54b013d5d105ecddea"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "cd511beb992d87e30b04347c8159352e"
server_key_iv: "b6e98040f01194cccaf98487dc81be81"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.090655 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.120584 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.123744 25713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:25.123991 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.131582 25713 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.125:43773
I20260924 20:20:25.131852 25780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.125:43773 every 8 connection(s)
I20260924 20:20:25.132685 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
I20260924 20:20:25.134884 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25713
I20260924 20:20:25.135759 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:25.140470 25781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.125@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:25.151783 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.136701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.18.172.125:43773)
0924 20:20:25.137114 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.137136 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.137174 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:20:25.137941 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.137962 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.138313 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.138611 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.139547 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.140834 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.141992 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.142642 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.142826 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.145773 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.145779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.145789 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.145817 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.148124 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.148679 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.148688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.148700 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.148839 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.149153 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.149155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.149157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.149729 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.149984 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.150270 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.150604 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260924 20:20:25.152459 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.142406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.18.172.126:34959)
0924 20:20:25.142566 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.142570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.142882 (+ 312us) server_negotiation.cc:407] Connection header received
0924 20:20:25.143191 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.143196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.143254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.143353 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.144226 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.145095 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.146065 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.146284 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.146393 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.148958 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.148962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.148964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.148997 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.150668 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.151208 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.151211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.151212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.151288 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.151581 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.151583 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.151584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.151785 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.151881 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.152249 (+ 368us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.152354 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:20:25.152546 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.142308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35256)
0924 20:20:25.142755 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.142808 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.142992 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.143510 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.143515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.143869 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.144096 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.144109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.145243 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.145247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.145928 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.145937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.146393 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.146397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.146505 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.146966 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.146994 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.148792 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.150769 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.150776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.150782 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.151114 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.151381 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.151385 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.151387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.151486 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.151900 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.151906 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.152037 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.152180 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.152363 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":140,"threads_started":1}
I20260924 20:20:25.153796 25781 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.125@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
I20260924 20:20:25.162308 25785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.154395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50270)
0924 20:20:25.154483 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.154494 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.154581 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.154792 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.154795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.154954 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.155095 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.155101 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.156271 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.156274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.156834 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.156841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.157278 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.157281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.157376 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.157870 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.157881 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.159166 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.161034 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.161037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.161038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.161212 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.161480 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.161482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.161484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.161557 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.161939 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.161940 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.162023 (+ 83us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.162102 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.162196 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 20:20:25.162451 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.154498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.18.172.125:43773)
0924 20:20:25.154593 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.154617 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.154631 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:25.154661 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.154664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.154696 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.154783 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.155194 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.156143 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.156945 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.157179 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.157282 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.159283 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.159286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.159288 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.159308 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.160920 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.161300 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.161303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.161304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.161384 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.161642 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.161645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.161646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.161851 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.161945 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.162180 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.162288 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:20:25.165642 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Bootstrap starting.
I20260924 20:20:25.168336 25781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:25.169183 25781 log.cc:824] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:25.171350 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: No bootstrap required, opened a new log
I20260924 20:20:25.175280 25781 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } }
I20260924 20:20:25.175479 25781 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:25.175523 25781 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7700bf492734b54b013d5d105ecddea, State: Initialized, Role: FOLLOWER
I20260924 20:20:25.175897 25781 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } }
I20260924 20:20:25.176292 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } } }
I20260924 20:20:25.176399 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:25.177008 25781 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:20:25.181463 25798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:20:25.181555 25798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:20:25.183490 25713 master_runner.cc:428] Detected that this master b7700bf492734b54b013d5d105ecddea is joining an existing cluster
I20260924 20:20:25.183560 25713 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.172.125:43773
I20260924 20:20:25.194515 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.186016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50276 (local address 127.18.172.125:43773)
0924 20:20:25.186132 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.186136 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.187277 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 20:20:25.187375 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.187377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.187415 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.187492 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.188015 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.188496 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.189767 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.190052 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.190157 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.191348 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.191352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.191354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.191375 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.192788 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193349 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.193352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.193354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.193426 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193693 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.193698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.193701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.193867 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.193920 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.194208 (+ 288us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.194320 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":48}
I20260924 20:20:25.194809 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.185697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35270 (local address 127.18.172.126:34959)
0924 20:20:25.185821 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.185826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.186115 (+ 289us) server_negotiation.cc:407] Connection header received
0924 20:20:25.186263 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.186266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.186332 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.186414 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.186850 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.187576 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.188760 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.189101 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.189199 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.190379 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.190386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.190391 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.190422 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.192247 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193360 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.193363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.193365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.193458 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193741 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.193746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.193749 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.193962 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.194061 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.194363 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.194481 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:25.194947 25804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.185752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50276)
0924 20:20:25.187185 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.187199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.187294 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.187584 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.187586 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.187801 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.187904 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.187909 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.188604 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.188607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.189401 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.189409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.190156 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.190159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.190261 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.190737 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.190748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.191244 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.192892 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.192897 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.192900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.193248 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.193503 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193504 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.193506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.193599 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.193960 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.193962 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.194057 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.194655 (+ 598us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.194765 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1371,"spinlock_wait_cycles":2304,"thread_start_us":105,"threads_started":1}
I20260924 20:20:25.195247 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.185667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35270)
0924 20:20:25.186040 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.186055 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.186156 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.186428 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.186430 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.186648 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.186737 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.186742 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.187666 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.187669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.188640 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.188650 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.189231 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.189235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.189327 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.189803 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.189818 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.190240 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.192910 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.192912 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.192914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.193248 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.193569 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.193571 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.193573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.193649 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.194065 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.194067 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.194163 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.194260 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.194373 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1,"spinlock_wait_cycles":1792,"thread_start_us":95,"threads_started":1}
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.126@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
I20260924 20:20:25.208935 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.199697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50282)
0924 20:20:25.200057 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.200106 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.200233 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.200534 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.200539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.200860 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.200957 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.200962 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.201914 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.201916 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.202559 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.202568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.203123 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.203126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.203237 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.203728 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.203739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.205234 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.207223 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.207235 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.207243 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.207494 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.207809 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.207812 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.207814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.207893 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.208324 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.208331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.208450 (+ 119us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.208588 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.208770 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":118,"threads_started":1}
I20260924 20:20:25.208950 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.199737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.18.172.125:43773)
0924 20:20:25.199854 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.199858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.200186 (+ 328us) server_negotiation.cc:407] Connection header received
0924 20:20:25.200360 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.200363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.200412 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.200493 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.201064 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.201807 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.202666 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.202995 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.203115 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.205341 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.205346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.205351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.205374 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.207088 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.207599 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.207604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.207606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.207688 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.208021 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.208026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.208030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.208213 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.208323 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.208660 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.208766 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:20:25.210283 25249 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.172.125:43773: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b7700bf492734b54b013d5d105ecddea"
member_type: NON_VOTER
last_known_addr {
host: "127.18.172.125"
port: 43773
}
attrs {
promote: true
}
}
dest_uuid: "6c03f18dff984dd893e3f7719b2526da"
cas_config_opid_index: -1
I20260924 20:20:25.210958 25249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } }
I20260924 20:20:25.212786 25808 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b7700bf492734b54b013d5d105ecddea (127.18.172.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } }
I20260924 20:20:25.213424 25755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:25.214273 25808 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.172.125:43773
I20260924 20:20:25.214485 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } }
I20260924 20:20:25.214572 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:25.214679 25713 master.cc:537] Master@127.18.172.125:43773 shutting down...
I20260924 20:20:25.214754 25755 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 20:20:25.215574 25809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:20:25.217754 25235 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
WARNING: no policy specified for oryx/127.18.172.124@KRBTEST.COM; defaulting to no policy
W20260924 20:20:25.218607 25235 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da -> Peer b7700bf492734b54b013d5d105ecddea (127.18.172.125:43773): Couldn't send request to peer b7700bf492734b54b013d5d105ecddea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 20:20:25.219381 25713 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:25.219650 25713 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:25.219758 25713 tablet_replica.cc:330] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.124@KRBTEST.COM" created.
I20260924 20:20:25.224383 25713 master.cc:552] Master@127.18.172.125:43773 shutdown complete.
I20260924 20:20:25.224443 25713 master_runner.cc:305] Clearing existing system tablet
I20260924 20:20:25.224680 25713 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.224745 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.225844 25713 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:25.226684 25814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.226886 25713 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.227059 25713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.227102 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "b7700bf492734b54b013d5d105ecddea"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "cd511beb992d87e30b04347c8159352e"
server_key_iv: "b6e98040f01194cccaf98487dc81be81"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.229068 25713 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:25.231171 25713 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:20:25.231276 25713 log.cc:1196] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 20:20:25.232046 25713 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Deleting consensus metadata
I20260924 20:20:25.232239 25713 master_runner.cc:315] Copying system tablet from 127.18.172.126:34959
I20260924 20:20:25.233453 25713 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: tablet copy: Beginning tablet copy session from remote peer at address 127.18.172.126:34959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.124.keytab.
Entry for principal oryx/127.18.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.124.keytab.
I20260924 20:20:25.239223 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:33793
--webserver_interface=127.18.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.124.keytab
--principal=oryx/127.18.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:20:25.242810 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.234038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35280)
0924 20:20:25.234281 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.234298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.234434 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.234889 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.234893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.235122 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.235239 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.235247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.236180 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.236184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.237092 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.237102 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.237638 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.237642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.237731 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.238335 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.238350 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.238792 (+ 442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.241226 (+ 2434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.241228 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.241230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.241428 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.241735 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.241739 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.241741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.241827 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.242231 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.242234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.242342 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.242447 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.242616 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":82,"threads_started":1}
I20260924 20:20:25.242801 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.234161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35280 (local address 127.18.172.126:34959)
0924 20:20:25.234314 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.234321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.234369 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:20:25.234588 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.234592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.234673 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.234782 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.235409 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.236035 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.237209 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.237521 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.237607 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.238957 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.238961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.238965 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.238997 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.241079 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.241542 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.241547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.241549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.241640 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.241920 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.241924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.241926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.242132 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.242243 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.242516 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.242627 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:20:25.243383 25269 tablet_copy_service.cc:140] P 6c03f18dff984dd893e3f7719b2526da: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b7700bf492734b54b013d5d105ecddea ({username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:35280)
I20260924 20:20:25.243573 25269 tablet_copy_service.cc:161] P 6c03f18dff984dd893e3f7719b2526da: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b7700bf492734b54b013d5d105ecddea at {username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:35280: session id = b7700bf492734b54b013d5d105ecddea-00000000000000000000000000000000
I20260924 20:20:25.245185 25269 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:20:25.246699 25713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:20:25.248667 25713 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: tablet copy: Starting download of 0 data blocks...
I20260924 20:20:25.248876 25713 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: tablet copy: Starting download of 1 WAL segments...
I20260924 20:20:25.250573 25713 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:20:25.252308 25269 tablet_copy_service.cc:342] P 6c03f18dff984dd893e3f7719b2526da: Request end of tablet copy session b7700bf492734b54b013d5d105ecddea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:35280
I20260924 20:20:25.252435 25269 tablet_copy_service.cc:433] P 6c03f18dff984dd893e3f7719b2526da: ending tablet copy session b7700bf492734b54b013d5d105ecddea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b7700bf492734b54b013d5d105ecddea
I20260924 20:20:25.259697 25713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:25.261509 25822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.261509 25825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.263217 25713 server_base.cc:1034] running on GCE node
W20260924 20:20:25.263361 25823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.263588 25713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.263907 25713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.265046 25713 hybrid_clock.cc:648] HybridClock initialized: now 1790281225265058 us; error 46 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.266538 25713 init.cc:377] Logged in from keytab as oryx/127.18.172.125@KRBTEST.COM (short username oryx)
I20260924 20:20:25.267009 25713 webserver.cc:460] Webserver started at http://127.18.172.125:34819/ using document root <none> and password file <none>
I20260924 20:20:25.267220 25713 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.267275 25713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.268347 25713 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.269263 25832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.269505 25713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.269598 25713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "b7700bf492734b54b013d5d105ecddea"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "cd511beb992d87e30b04347c8159352e"
server_key_iv: "b6e98040f01194cccaf98487dc81be81"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.269682 25713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.280588 25713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.282960 25713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.288286 25713 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.125:43773
I20260924 20:20:25.288828 25713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
I20260924 20:20:25.290266 25885 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:20:25.291191 25885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Bootstrap starting.
I20260924 20:20:25.299119 25885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:20:25.299414 25885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Bootstrap complete.
I20260924 20:20:25.299448 25884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.125:43773 every 8 connection(s)
I20260924 20:20:25.299993 25885 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } }
I20260924 20:20:25.300127 25885 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b7700bf492734b54b013d5d105ecddea, State: Initialized, Role: LEARNER
I20260924 20:20:25.300241 25885 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } }
I20260924 20:20:25.300580 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } }
I20260924 20:20:25.300740 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: LEARNER
I20260924 20:20:25.300747 25885 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:25.303651 25899 catalog_manager.cc:1259] Loaded cluster ID: ff1caf86e16841febce96b6acb2c6866
I20260924 20:20:25.303750 25899 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: loading cluster ID for follower catalog manager: success
I20260924 20:20:25.305507 25899 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: acquiring CA information for follower catalog manager: success
I20260924 20:20:25.306563 25899 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 20:20:25.387768 25821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:25.388046 25821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:25.388087 25821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:25.392122 25821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:25.392180 25821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:25.392199 25821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:25.392218 25821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:25.392233 25821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:25.399114 25821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.124.keytab
--principal=oryx/127.18.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:33793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:25.400575 25821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:25.402019 25821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:25.410817 25905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.411057 25821 server_base.cc:1034] running on GCE node
W20260924 20:20:25.410861 25904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.410874 25907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.411715 25821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.412969 25821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.414173 25821 hybrid_clock.cc:648] HybridClock initialized: now 1790281225414147 us; error 40 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.417030 25821 init.cc:377] Logged in from keytab as oryx/127.18.172.124@KRBTEST.COM (short username oryx)
I20260924 20:20:25.417902 25821 webserver.cc:460] Webserver started at http://127.18.172.124:46311/ using document root <none> and password file <none>
I20260924 20:20:25.418408 25821 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.418464 25821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.418669 25821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:25.420769 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/instance:
uuid: "df9878cf9f114bf3822e07e1e7b3230d"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "7e09de483f857861e6c12827f3c73e52"
server_key_iv: "c0d891cf9355bdf3223f98e201287469"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.421254 25821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal/instance:
uuid: "df9878cf9f114bf3822e07e1e7b3230d"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "7e09de483f857861e6c12827f3c73e52"
server_key_iv: "c0d891cf9355bdf3223f98e201287469"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.424923 25821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260924 20:20:25.427299 25914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.428740 25821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:20:25.428838 25821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "df9878cf9f114bf3822e07e1e7b3230d"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "7e09de483f857861e6c12827f3c73e52"
server_key_iv: "c0d891cf9355bdf3223f98e201287469"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.428933 25821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.455446 25821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.458880 25821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:25.459131 25821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.462502 25440 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
I20260924 20:20:25.466593 25821 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.124:33793
I20260924 20:20:25.466895 25966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.124:33793 every 8 connection(s)
I20260924 20:20:25.467751 25821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
I20260924 20:20:25.467957 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25821
I20260924 20:20:25.470949 25967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:25.475808 25967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
I20260924 20:20:25.484498 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.469468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.18.172.124:33793)
0924 20:20:25.469925 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.469948 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.469990 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:20:25.470840 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.470863 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.471336 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.471709 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.472397 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.474079 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.474966 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.475658 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.475816 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.478922 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.478927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.478939 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.478965 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.481068 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.481640 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.481645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.481649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.481754 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.482042 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.482045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.482046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.482514 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.482717 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.483102 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.483441 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":100,"threads_started":1}
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.124@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:25.488559 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.478107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35282 (local address 127.18.172.126:34959)
0924 20:20:25.478232 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.478236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.479160 (+ 924us) server_negotiation.cc:407] Connection header received
0924 20:20:25.479329 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.479332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.479376 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.479553 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.480336 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.481129 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.482290 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.482615 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.482715 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.485400 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.485404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.485407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.485435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.487116 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.487518 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.487520 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.487522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.487603 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.487828 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.487830 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.487831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.487997 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.488073 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.488341 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.488435 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:25.488566 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.477992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35282)
0924 20:20:25.478985 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.479084 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.479212 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.479602 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.479608 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.480068 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.480211 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.480217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.481237 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.481240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.482184 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.482193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.482728 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.482732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.482924 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.483583 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.483596 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.485245 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.487211 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.487218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.487222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.487433 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.487680 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.487682 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.487683 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.487755 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.488082 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.488087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.488179 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.488287 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.488418 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":174,"threads_started":1}
I20260924 20:20:25.489571 25967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.124@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
I20260924 20:20:25.497710 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.489995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50298)
0924 20:20:25.490107 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.490119 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.490177 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.490597 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.490599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.490761 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.490848 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.490853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.491804 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.491811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.492368 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.492375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.492841 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.492845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.492942 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.493375 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.493386 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.494602 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.496236 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.496246 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.496248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.496437 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.496713 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.496717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.496719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.496801 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.497176 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.497179 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.497277 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.497394 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.497532 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260924 20:20:25.497722 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.490128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50298 (local address 127.18.172.125:43773)
0924 20:20:25.490391 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.490394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.490406 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:25.490440 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.490442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.490478 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.490579 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.490964 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.491696 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.492498 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.492730 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.492843 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.494742 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.494745 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.494748 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.494780 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.496125 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.496535 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.496540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.496542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.496621 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.496889 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.496894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.496896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.497081 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.497187 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.497454 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.497575 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260924 20:20:25.498549 25967 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.124@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
I20260924 20:20:25.506440 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.498957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39884)
0924 20:20:25.499126 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.499141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.499221 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.499424 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.499427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.499592 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.499648 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.499653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.500710 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.500716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.501285 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.501292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.501781 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.501785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.501857 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.502238 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.502249 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.503490 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.505077 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.505080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.505081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.505250 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.505619 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.505622 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.505623 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.505694 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.506089 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.506091 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.506161 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.506238 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.506327 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 20:20:25.506763 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.499058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.18.172.124:33793)
0924 20:20:25.499129 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.499136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.499184 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:20:25.499287 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.499292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.499331 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.499414 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.499754 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.500592 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.501397 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.501681 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.501785 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.503611 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.503613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.503615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.503635 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.504983 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.505437 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.505440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.505441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.505517 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.505810 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.505813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.505814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.505996 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.506083 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.506461 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.506611 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 20:20:25.509722 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Bootstrap starting.
I20260924 20:20:25.512068 25967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:25.512805 25967 log.cc:824] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:25.515077 25967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: No bootstrap required, opened a new log
I20260924 20:20:25.518122 25967 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } }
I20260924 20:20:25.518383 25967 raft_consensus.cc:374] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:25.518465 25967 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df9878cf9f114bf3822e07e1e7b3230d, State: Initialized, Role: FOLLOWER
I20260924 20:20:25.518931 25967 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } }
I20260924 20:20:25.519488 25973 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } } }
I20260924 20:20:25.519613 25973 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:25.520215 25967 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: configured and running, proceeding with master startup.
W20260924 20:20:25.525098 25984 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:20:25.525189 25984 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:20:25.528812 25821 master_runner.cc:428] Detected that this master df9878cf9f114bf3822e07e1e7b3230d is joining an existing cluster
I20260924 20:20:25.528896 25821 master_runner.cc:432] Initiating AddMaster RPC to add 127.18.172.124:33793
I20260924 20:20:25.537895 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25311
I20260924 20:20:25.543498 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:43265
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:34959
--webserver_port=41965
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:20:25.564637 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.530935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35286)
0924 20:20:25.531530 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.531544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.531695 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.532087 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.532090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.532379 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.532619 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.532627 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533733 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.533735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.535679 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.535685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.536325 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.536513 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.537299 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.537311 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.559765 (+ 22454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.561925 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.561929 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.561931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.562209 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.563487 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.563491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.563493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.563574 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.564034 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.564037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.564138 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.564236 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.564416 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":626,"spinlock_wait_cycles":6912,"thread_start_us":117,"threads_started":1}
I20260924 20:20:25.565696 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.531408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39896)
0924 20:20:25.531961 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.531974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.532085 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.532379 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.532381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.532555 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.532714 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.532724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533798 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.533801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.535675 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.535685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.536325 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.536518 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.559728 (+ 23210us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.559748 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.560212 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.562903 (+ 2691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.562906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.562908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.563153 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.564561 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.564564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.564566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.564635 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.565154 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.565157 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.565233 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.565329 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.565457 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":351,"thread_start_us":66,"threads_started":1}
I20260924 20:20:25.565892 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.531181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.18.172.126:34959)
0924 20:20:25.531339 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.531345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.531622 (+ 277us) server_negotiation.cc:407] Connection header received
0924 20:20:25.531813 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.531817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.531869 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.531931 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.532754 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533584 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.535806 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.536131 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536198 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.560029 (+ 23831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.560033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.560036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.560072 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.561788 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.563283 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.563287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.563289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.563379 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.563669 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.563672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.563674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.563906 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.563993 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.565625 (+ 1632us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.565743 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:20:25.567432 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.531088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50308 (local address 127.18.172.125:43773)
0924 20:20:25.531218 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.531222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.531659 (+ 437us) server_negotiation.cc:407] Connection header received
0924 20:20:25.531876 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.531880 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.531936 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.531990 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.532704 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533544 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.535632 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.535998 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536082 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.562303 (+ 26221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.562308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.562311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.562343 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.564104 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.565321 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.565325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.565327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.565412 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.566465 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.566469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.566470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.566685 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.566763 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.567140 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.567279 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:20:25.568444 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.531669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39896 (local address 127.18.172.124:33793)
0924 20:20:25.531770 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.531776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.532028 (+ 252us) server_negotiation.cc:407] Connection header received
0924 20:20:25.532169 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.532172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.532208 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.532253 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.532931 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533641 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.535797 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.536115 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536188 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.560638 (+ 24450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.560641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.560644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.560669 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.562793 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.564385 (+ 1592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.564389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.564391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.564459 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.564711 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.564714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.564715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.564887 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.564966 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.568188 (+ 3222us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.568295 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 20:20:25.568382 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.531243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50308)
0924 20:20:25.531576 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.531589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.531723 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.532132 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.532135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.532397 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.532556 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.532564 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.533725 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.533728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.535498 (+ 1770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.535508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.536253 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.536257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.536391 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.536885 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.537083 (+ 198us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.561081 (+ 23998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.564249 (+ 3168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.564252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.564255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.565200 (+ 945us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.565515 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.565518 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.565520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.566348 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.566821 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.566824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.566908 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.567814 (+ 906us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.567953 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":59,"thread_start_us":101,"threads_started":1}
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281224, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.126@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
I20260924 20:20:25.580277 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.571171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.18.172.124:33793)
0924 20:20:25.571271 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.571275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.571291 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:25.571458 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.571461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.571518 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.571598 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.572040 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.572819 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.574013 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.574322 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.574423 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.576862 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.576866 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.576869 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.576894 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.578498 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.579038 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.579042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.579044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.579127 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.579409 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.579412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.579414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.579638 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.579734 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.580023 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.580134 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:20:25.580278 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.571055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39908)
0924 20:20:25.571244 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.571258 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.571335 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.571632 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.571635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.571821 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.571928 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.571934 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.572942 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.572945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.573884 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.573896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.574443 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.574446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.574595 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.575136 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.575150 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.576715 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.578612 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.578616 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.578618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.578895 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.579219 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.579222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.579224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.579316 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.579736 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.579739 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.579853 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.579961 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.580106 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 20:20:25.581861 25249 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.18.172.124:33793: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d"
member_type: NON_VOTER
last_known_addr {
host: "127.18.172.124"
port: 33793
}
attrs {
promote: true
}
}
dest_uuid: "6c03f18dff984dd893e3f7719b2526da"
cas_config_opid_index: 5
I20260924 20:20:25.582489 25249 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } }
I20260924 20:20:25.583580 25809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.585062 25809 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.18.172.124:33793
I20260924 20:20:25.585235 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.585330 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:25.586452 25941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:25.587106 25821 master.cc:537] Master@127.18.172.124:33793 shutting down...
I20260924 20:20:25.587854 25941 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:20:25.588512 25808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:20:25.590147 25234 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da -> Peer df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793): Couldn't send request to peer df9878cf9f114bf3822e07e1e7b3230d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 20:20:25.593055 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.585757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.18.172.125:43773)
0924 20:20:25.585890 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.585896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.587782 (+ 1886us) server_negotiation.cc:407] Connection header received
0924 20:20:25.588126 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.588129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.588196 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.588272 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:25.588793 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.589596 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.591511 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.592115 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.592207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.592339 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.592738 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.592913 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:20:25.593647 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.585621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:50320)
0924 20:20:25.587704 (+ 2083us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.587719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.587990 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.588453 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.588456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.588474 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:25.588575 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.588581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.589797 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.589801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.591356 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.591367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.592439 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.592442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.592545 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.592589 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.593430 (+ 841us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.593522 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2002}
I20260924 20:20:25.593997 25821 raft_consensus.cc:2229] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:25.594272 25821 raft_consensus.cc:2258] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:25.594297 25851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 LEARNER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 20:20:25.594362 25821 tablet_replica.cc:330] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: stopping tablet replica
I20260924 20:20:25.594707 25808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 20:20:25.596611 25851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.599133 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c03f18dff984dd893e3f7719b2526da. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.599246 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: LEARNER
I20260924 20:20:25.599345 25821 master.cc:552] Master@127.18.172.124:33793 shutdown complete.
I20260924 20:20:25.599331 25888 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: NON_VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: true } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.599383 25821 master_runner.cc:305] Clearing existing system tablet
I20260924 20:20:25.599390 25888 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: LEARNER
I20260924 20:20:25.599617 25821 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.599670 25821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.600916 25821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:25.601881 25997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.602111 25821 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.602357 25821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.602432 25821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "df9878cf9f114bf3822e07e1e7b3230d"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "7e09de483f857861e6c12827f3c73e52"
server_key_iv: "c0d891cf9355bdf3223f98e201287469"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.604434 25808 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: attempting to promote NON_VOTER b7700bf492734b54b013d5d105ecddea to VOTER
I20260924 20:20:25.604892 25808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } }
I20260924 20:20:25.606210 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.605913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39916)
0924 20:20:25.606047 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.606131 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:33793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:20:25.606622 25851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 LEARNER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 20:20:25.606972 25809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260924 20:20:25.607568 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.607309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39928)
0924 20:20:25.607439 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.607501 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:33793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 20:20:25.607940 25821 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:20:25.608165 25234 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da -> Peer df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793): Couldn't send request to peer df9878cf9f114bf3822e07e1e7b3230d. Status: Network error: Client connection negotiation failed: client connection to 127.18.172.124:33793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:20:25.608985 25809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b7700bf492734b54b013d5d105ecddea (127.18.172.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.610378 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.610502 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:25.610618 25809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.610690 25809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:25.611241 25821 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 20:20:25.611369 25821 log.cc:1196] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 20:20:25.611438 25851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b7700bf492734b54b013d5d105ecddea (127.18.172.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.611775 25821 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Deleting consensus metadata
I20260924 20:20:25.611981 25821 master_runner.cc:315] Copying system tablet from 127.18.172.126:34959
I20260924 20:20:25.613163 25821 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: tablet copy: Beginning tablet copy session from remote peer at address 127.18.172.126:34959
I20260924 20:20:25.613309 25995 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.613435 25995 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:25.627838 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.615318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:35296)
0924 20:20:25.619171 (+ 3853us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.619184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.619329 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.619629 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.619632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.619838 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.619958 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.619965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.621052 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.621056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.622132 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.622143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.622721 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.622725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.622819 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.623455 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.623477 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.623915 (+ 438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.625953 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.625957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.625959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.626208 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.626508 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.626511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.626513 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.626666 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.627169 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.627175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.627283 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.627393 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.627615 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3782,"thread_start_us":92,"threads_started":1}
I20260924 20:20:25.627852 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.615396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35296 (local address 127.18.172.126:34959)
0924 20:20:25.615515 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.615520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.619271 (+ 3751us) server_negotiation.cc:407] Connection header received
0924 20:20:25.619432 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.619435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.619492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.619570 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.620096 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.620929 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.622254 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.622597 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.622705 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.624071 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.624075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.624078 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.624107 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.625828 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.626312 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.626315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.626317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.626407 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.626769 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.626774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.626777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.627027 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.627120 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.627522 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.627689 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:20:25.628917 25269 tablet_copy_service.cc:140] P 6c03f18dff984dd893e3f7719b2526da: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer df9878cf9f114bf3822e07e1e7b3230d ({username='oryx', principal='oryx/127.18.172.124@KRBTEST.COM'} at 127.0.0.1:35296)
I20260924 20:20:25.629034 25269 tablet_copy_service.cc:161] P 6c03f18dff984dd893e3f7719b2526da: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer df9878cf9f114bf3822e07e1e7b3230d at {username='oryx', principal='oryx/127.18.172.124@KRBTEST.COM'} at 127.0.0.1:35296: session id = df9878cf9f114bf3822e07e1e7b3230d-00000000000000000000000000000000
I20260924 20:20:25.630151 25269 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 20:20:25.631577 25821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 20:20:25.633966 25821 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: tablet copy: Starting download of 0 data blocks...
I20260924 20:20:25.634150 25821 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: tablet copy: Starting download of 1 WAL segments...
I20260924 20:20:25.635823 25821 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 20:20:25.637660 25269 tablet_copy_service.cc:342] P 6c03f18dff984dd893e3f7719b2526da: Request end of tablet copy session df9878cf9f114bf3822e07e1e7b3230d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.18.172.124@KRBTEST.COM'} at 127.0.0.1:35296
I20260924 20:20:25.637753 25269 tablet_copy_service.cc:433] P 6c03f18dff984dd893e3f7719b2526da: ending tablet copy session df9878cf9f114bf3822e07e1e7b3230d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer df9878cf9f114bf3822e07e1e7b3230d
I20260924 20:20:25.645494 25821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:25.648434 26005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.652256 26006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.652318 26008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.653926 25821 server_base.cc:1034] running on GCE node
I20260924 20:20:25.654274 25821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.654654 25821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.655857 25821 hybrid_clock.cc:648] HybridClock initialized: now 1790281225655844 us; error 32 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.657621 25821 init.cc:377] Logged in from keytab as oryx/127.18.172.124@KRBTEST.COM (short username oryx)
I20260924 20:20:25.658046 25821 webserver.cc:460] Webserver started at http://127.18.172.124:34281/ using document root <none> and password file <none>
I20260924 20:20:25.658221 25821 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.658252 25821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.659585 25821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.660640 26015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.660907 25821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:25.660982 25821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "df9878cf9f114bf3822e07e1e7b3230d"
format_stamp: "Formatted at 2026-09-24 20:20:25 on dist-test-slave-m1qq"
server_key: "7e09de483f857861e6c12827f3c73e52"
server_key_iv: "c0d891cf9355bdf3223f98e201287469"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.661067 25821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.677911 25574 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
I20260924 20:20:25.678303 25821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.680493 25821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.685565 25821 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.124:33793
I20260924 20:20:25.686211 25821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
I20260924 20:20:25.686594 26070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.124:33793 every 8 connection(s)
I20260924 20:20:25.688048 26071 sys_catalog.cc:263] Verifying existing consensus state
W20260924 20:20:25.688390 25993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:25.688720 25993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:25.688795 25993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:20:25.689092 26071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Bootstrap starting.
W20260924 20:20:25.694624 25993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:25.694697 25993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:25.694792 25993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:25.700018 26071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:20:25.700320 26071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: Bootstrap complete.
I20260924 20:20:25.700901 26071 raft_consensus.cc:348] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } }
I20260924 20:20:25.701040 26071 raft_consensus.cc:729] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: df9878cf9f114bf3822e07e1e7b3230d, State: Initialized, Role: LEARNER
I20260924 20:20:25.701130 26071 consensus_queue.cc:260] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } }
I20260924 20:20:25.701447 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:25.701599 26071 sys_catalog.cc:564] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:25.701645 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: LEARNER
I20260924 20:20:25.702631 25993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:43265
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=41965
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:25.704146 25993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:25.705268 25993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:25.705274 26085 catalog_manager.cc:1259] Loaded cluster ID: ff1caf86e16841febce96b6acb2c6866
I20260924 20:20:25.705329 26085 catalog_manager.cc:1552] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: loading cluster ID for follower catalog manager: success
I20260924 20:20:25.707607 26085 catalog_manager.cc:1574] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: acquiring CA information for follower catalog manager: success
I20260924 20:20:25.708873 26085 catalog_manager.cc:1602] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 20:20:25.715197 26090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.715197 26087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.715200 26088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.715538 25993 server_base.cc:1034] running on GCE node
I20260924 20:20:25.715952 25993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.716600 25993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.717787 25993 hybrid_clock.cc:648] HybridClock initialized: now 1790281225717771 us; error 34 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.720567 25993 init.cc:377] Logged in from keytab as oryx/127.18.172.65@KRBTEST.COM (short username oryx)
I20260924 20:20:25.721441 25993 webserver.cc:460] Webserver started at http://127.18.172.65:41965/ using document root <none> and password file <none>
I20260924 20:20:25.722021 25993 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.722131 25993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.726150 25993 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:25.728595 26097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.729914 25993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:20:25.730051 25993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fb843e144828415e8953c52fb2213bb5"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "18fed9ce3ca0bccabfc3597acc58acb6"
server_key_iv: "f5a928e225539306ee8191809b52c582"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.730660 25993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.742163 25993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.745361 25993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:25.745586 25993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.746217 25993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:25.747176 25993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:25.747244 25993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.747311 25993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:25.747365 25993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.756268 25993 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:43265
I20260924 20:20:25.756289 26210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:43265 every 8 connection(s)
I20260924 20:20:25.757301 25993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:25.763232 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 25993
I20260924 20:20:25.763420 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25445
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
I20260924 20:20:25.769836 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:41779
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:34959
--webserver_port=44111
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:20:25.774282 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.759143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:44273 (local address 127.18.172.126:34959)
0924 20:20:25.759418 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.759422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.760159 (+ 737us) server_negotiation.cc:407] Connection header received
0924 20:20:25.761476 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.761480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.761539 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.761619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.763216 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.764047 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766346 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.766593 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766713 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.769887 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.769894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.769899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.769928 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.772009 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772500 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.772504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.772506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.772586 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772909 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.772913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.772914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.773125 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.773207 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.773987 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.774115 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187}
I20260924 20:20:25.774281 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.759128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:35943 (local address 127.18.172.125:43773)
0924 20:20:25.759302 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.759307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.760162 (+ 855us) server_negotiation.cc:407] Connection header received
0924 20:20:25.761483 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.761486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.761559 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.761615 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.763217 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.764044 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766329 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.766682 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766786 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.769937 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.769941 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.769944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.769970 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.771765 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772413 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.772418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.772420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.772507 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772916 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.772920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.772921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.773125 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.773207 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.773995 (+ 788us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.774115 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
I20260924 20:20:25.776037 26216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.759518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.18.172.65:35943)
0924 20:20:25.760011 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.760063 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.761266 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.761801 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.761808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.762460 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.763046 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.763066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.764794 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.764797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.766198 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.766209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.766914 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.767064 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.767634 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.767653 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.769790 (+ 2137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.771899 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.771905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.771930 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.772291 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.772618 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.772626 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.772815 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.773277 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.773283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.773680 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.774439 (+ 759us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.774885 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":365,"spinlock_wait_cycles":46592,"thread_start_us":133,"threads_started":1}
I20260924 20:20:25.776343 26215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.759518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:44273)
0924 20:20:25.760010 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.760062 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.761266 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.761800 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.761807 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.762460 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.763046 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.763066 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.764552 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.764558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.766199 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.766209 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.766775 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.766780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.766952 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.767779 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.767790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.769665 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.772129 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772133 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.772135 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.772375 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.772687 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.772690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.772691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.772796 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.773277 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.773283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.773680 (+ 397us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.774481 (+ 801us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.774932 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":385,"spinlock_wait_cycles":96128,"thread_start_us":144,"threads_started":1}
I20260924 20:20:25.777431 26212 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:25.777688 26212 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.778227 26212 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:25.777431 26213 heartbeater.cc:347] Connected to a master server at 127.18.172.125:43773
I20260924 20:20:25.778868 26213 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.779237 26213 heartbeater.cc:511] Master 127.18.172.125:43773 requested a full tablet report, sending...
I20260924 20:20:25.779604 25249 ts_manager.cc:194] Re-registered known tserver with Master: fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265)
I20260924 20:20:25.780252 26217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.760562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.18.172.65:41179)
0924 20:20:25.760831 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.760845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.761266 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.761804 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.761809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.762417 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.763055 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.763068 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.770328 (+ 7260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.770334 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.771786 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.771798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.772615 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.772620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.772745 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.773255 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.773272 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.775319 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.778247 (+ 2928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.778250 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.778252 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.778537 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.778900 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.778905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.778909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.779040 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.779608 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.779614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.779725 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.779832 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.779994 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":28,"spinlock_wait_cycles":1408,"thread_start_us":96,"threads_started":1}
I20260924 20:20:25.780261 25849 ts_manager.cc:194] Registered new tserver with Master: fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265)
I20260924 20:20:25.780319 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:44273
I20260924 20:20:25.780920 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.760720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:41179 (local address 127.18.172.124:33793)
0924 20:20:25.760968 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.760972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.760986 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:25.761560 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.761564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.761617 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.761690 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.763217 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.770198 (+ 6981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.771922 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.772324 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.772439 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.775497 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.775504 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.775507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.775543 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.778120 (+ 2577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.778660 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.778667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.778670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.778777 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.779163 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.779169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.779172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.779437 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.779542 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.780509 (+ 967us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.780676 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
I20260924 20:20:25.782294 26211 heartbeater.cc:347] Connected to a master server at 127.18.172.124:33793
I20260924 20:20:25.782397 26211 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.782611 26211 heartbeater.cc:511] Master 127.18.172.124:33793 requested a full tablet report, sending...
I20260924 20:20:25.783648 26035 ts_manager.cc:194] Registered new tserver with Master: fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265)
W20260924 20:20:25.888764 26219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:25.889042 26219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:25.889133 26219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:25.893200 26219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:25.893288 26219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:25.893374 26219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:25.898664 26219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:41779
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=44111
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:25.899772 26219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:25.900858 26219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:25.909719 25708 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
W20260924 20:20:25.910843 26224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:25.910858 26225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.911091 26219 server_base.cc:1034] running on GCE node
W20260924 20:20:25.911801 26227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:25.912246 26219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:25.912818 26219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:25.914009 26219 hybrid_clock.cc:648] HybridClock initialized: now 1790281225913993 us; error 30 us; skew 500 ppm
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:25.916673 26219 init.cc:377] Logged in from keytab as oryx/127.18.172.66@KRBTEST.COM (short username oryx)
I20260924 20:20:25.917518 26219 webserver.cc:460] Webserver started at http://127.18.172.66:44111/ using document root <none> and password file <none>
I20260924 20:20:25.918080 26219 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:25.918185 26219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:25.922192 26219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:25.924360 26234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.925657 26219 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:25.925796 26219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "d5d7e03495e748f28e6c45ab824b6c98"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "cf138f885595711df3ae7b06f46bfd20"
server_key_iv: "b722b9481e6e488d0ff03e9a63aac052"
server_key_version: "encryptionkey@0"
I20260924 20:20:25.926342 26219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:25.944826 26219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:25.947801 26219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:25.948030 26219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:25.948619 26219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:25.949465 26219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:25.949539 26219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.949612 26219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:25.949656 26219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:25.959203 26219 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:41779
I20260924 20:20:25.959220 26347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:41779 every 8 connection(s)
I20260924 20:20:25.960215 26219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:25.969020 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 26219
I20260924 20:20:25.969770 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25579
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
Sep 24 20:20:25 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281225, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
I20260924 20:20:25.976223 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.963784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:34645 (local address 127.18.172.126:34959)
0924 20:20:25.963910 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.963914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.963986 (+ 72us) server_negotiation.cc:407] Connection header received
0924 20:20:25.964824 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.964836 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.964891 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.964996 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.966072 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.966869 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968264 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968611 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.971534 (+ 2850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.971537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.971540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.971566 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.973140 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.973800 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.973807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.973812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.973916 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.974836 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.974840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.974842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.975128 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.975203 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.975904 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.976039 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:25.976693 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.962099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:43389 (local address 127.18.172.125:43773)
0924 20:20:25.962241 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.962246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.963261 (+ 1015us) server_negotiation.cc:407] Connection header received
0924 20:20:25.964379 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.964383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.964564 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.964690 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.966060 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.966589 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.967815 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968041 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968152 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.972907 (+ 4755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.972912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.972915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.972944 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.974909 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975388 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.975391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.975393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.975515 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975809 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.975814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.975816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.976026 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.976106 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.976428 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.976558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:20:25.977407 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.962106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48013 (local address 127.18.172.124:33793)
0924 20:20:25.962352 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:25.962467 (+ 115us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:25.963488 (+ 1021us) server_negotiation.cc:407] Connection header received
0924 20:20:25.964370 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:25.964373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:25.964433 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:25.964492 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:25.966073 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.966868 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968264 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968611 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968684 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.973309 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:25.973315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:25.973321 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:25.973351 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:25.975385 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975921 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.975928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.975932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.976020 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:25.976329 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:25.976334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:25.976339 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:25.976574 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:25.976650 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:25.977009 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 20:20:25.977158 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260924 20:20:25.977967 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.962526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.18.172.66:43389)
0924 20:20:25.963078 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.963141 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.964155 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.964751 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.964756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.965312 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.965911 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.965924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.966740 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.966743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.967689 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.967698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968190 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.968409 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.969082 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.969115 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.972739 (+ 3624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.975046 (+ 2307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975049 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.975051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.975286 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.975625 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975628 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.975630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.975706 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.976161 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.976164 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.976245 (+ 81us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.976880 (+ 635us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.976993 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":7,"spinlock_wait_cycles":8320,"thread_start_us":595,"threads_started":1}
I20260924 20:20:25.978068 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.962527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.18.172.66:48013)
0924 20:20:25.963399 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.963413 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.964160 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.964748 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.964755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.965331 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.965925 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.965934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.967029 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.967033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.968127 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.968137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968837 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.969027 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.969668 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.969681 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.973153 (+ 3472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.975511 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.975518 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.975523 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.975796 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.976130 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.976136 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.976141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.976231 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.976705 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.976710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.976817 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.977613 (+ 796us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.977758 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":325,"spinlock_wait_cycles":23808,"thread_start_us":150,"threads_started":1}
I20260924 20:20:25.978236 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:25.963662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:34645)
0924 20:20:25.963915 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:25.963929 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:25.964169 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:25.965031 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:25.965035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:25.965331 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:25.965929 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.965934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.967029 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.967033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.968127 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:25.968137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:25.968845 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:25.968848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:25.969006 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:25.969607 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:25.969639 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:25.971366 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:25.973273 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.973281 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.973304 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.973669 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.974243 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:25.974249 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:25.974254 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:25.974707 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:25.975286 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:25.975293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:25.975625 (+ 332us) client_negotiation.cc:769] Sending connection context
0924 20:20:25.976456 (+ 831us) client_negotiation.cc:240] Negotiation successful
0924 20:20:25.976801 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":265,"spinlock_wait_cycles":23808,"thread_start_us":90,"threads_started":1}
I20260924 20:20:25.979480 26350 heartbeater.cc:347] Connected to a master server at 127.18.172.125:43773
I20260924 20:20:25.979725 26350 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.979920 26348 heartbeater.cc:347] Connected to a master server at 127.18.172.124:33793
I20260924 20:20:25.979954 26349 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:25.979995 26348 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.980047 26349 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:25.980351 26350 heartbeater.cc:511] Master 127.18.172.125:43773 requested a full tablet report, sending...
I20260924 20:20:25.980352 26348 heartbeater.cc:511] Master 127.18.172.124:33793 requested a full tablet report, sending...
I20260924 20:20:25.980460 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:35541
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:34959
--webserver_port=46233
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:20:25.980993 26349 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:25.981576 26035 ts_manager.cc:194] Registered new tserver with Master: d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779)
I20260924 20:20:25.981602 25849 ts_manager.cc:194] Registered new tserver with Master: d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779)
I20260924 20:20:25.981714 25249 ts_manager.cc:194] Re-registered known tserver with Master: d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779)
I20260924 20:20:25.982398 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:34645
I20260924 20:20:26.026275 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.022161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.18.172.124:33793)
0924 20:20:26.022296 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.022300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.022315 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:20:26.022504 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.022507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.022558 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.022640 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.022990 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.023545 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.025180 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.025641 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.025783 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.025893 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.026090 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.026169 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:20:26.026292 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.022081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:39940)
0924 20:20:26.022238 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.022258 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.022382 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.022681 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.022684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.022704 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.022818 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.022825 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.023672 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.023676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.025029 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.025043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.025818 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.025823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.025932 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.025961 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.026053 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.026141 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:20:26.027060 26045 raft_consensus.cc:1203] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 20:20:26.027484 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c03f18dff984dd893e3f7719b2526da. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:26.027654 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: LEARNER
W20260924 20:20:26.106899 26355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:26.107264 26355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:26.107332 26355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:26.112254 26355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:26.112316 26355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:26.112430 26355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:26.118788 26355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:35541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=46233
--tserver_master_addrs=127.18.172.126:34959,127.18.172.125:43773,127.18.172.124:33793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:26.119982 26355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:26.121102 26355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:26.131834 26363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:26.131834 26361 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:26.131867 26360 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:26.132383 26355 server_base.cc:1034] running on GCE node
I20260924 20:20:26.132777 26355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:26.133468 26355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:26.134624 26355 hybrid_clock.cc:648] HybridClock initialized: now 1790281226134605 us; error 32 us; skew 500 ppm
Sep 24 20:20:26 dist-test-slave-m1qq krb5kdc[25204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281226, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:26.137681 26355 init.cc:377] Logged in from keytab as oryx/127.18.172.67@KRBTEST.COM (short username oryx)
I20260924 20:20:26.138710 26355 webserver.cc:460] Webserver started at http://127.18.172.67:46233/ using document root <none> and password file <none>
I20260924 20:20:26.139410 26355 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:26.139518 26355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:26.144104 26355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:26.146471 26370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:26.147881 26355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:20:26.148027 26355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "8edad0b4304c42998162c48e774d2a0d"
format_stamp: "Formatted at 2026-09-24 20:20:24 on dist-test-slave-m1qq"
server_key: "4ba85f566a1c3c9de2dfb4852af692da"
server_key_iv: "8ea6807ab19f35f7134206206435a8ea"
server_key_version: "encryptionkey@0"
I20260924 20:20:26.148674 26355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:26.159200 26355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:26.163093 26355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:26.163270 26355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:26.163903 26355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:26.164880 26355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:26.164937 26355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:26.164985 26355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:26.165005 26355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:26.175240 26355 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:35541
I20260924 20:20:26.175330 26483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:35541 every 8 connection(s)
I20260924 20:20:26.176278 26355 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:26.178634 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 26355
Sep 24 20:20:26 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281226, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.125@KRBTEST.COM
Sep 24 20:20:26 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281226, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.124@KRBTEST.COM
Sep 24 20:20:26 dist-test-slave-m1qq krb5kdc[25204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281226, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:26.194727 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.178234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:39827 (local address 127.18.172.125:43773)
0924 20:20:26.178369 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.178374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.179510 (+ 1136us) server_negotiation.cc:407] Connection header received
0924 20:20:26.180703 (+ 1193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.180707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.180763 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.180830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.182607 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.183393 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185134 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.185513 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185605 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.190135 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.190139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.190142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.190168 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.192157 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.192832 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.192836 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.192838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.192924 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193380 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.193384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.193385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.193600 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.193681 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.194451 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.194565 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":52}
I20260924 20:20:26.194731 25288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.178234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:33975 (local address 127.18.172.126:34959)
0924 20:20:26.178369 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.178374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.180345 (+ 1971us) server_negotiation.cc:407] Connection header received
0924 20:20:26.180723 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.180727 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.180785 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.180843 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.182893 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.183701 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185140 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.185498 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185587 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.190300 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.190305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.190308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.190332 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.192062 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.192841 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.192847 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.192849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.192938 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193401 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.193405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.193406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.193640 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.193713 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.194457 (+ 744us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.194581 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:26.194635 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.180987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50332 (local address 127.18.172.125:43773)
0924 20:20:26.181596 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.181601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.181613 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:26.181652 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.181657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.181704 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.181783 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.182914 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.183925 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185197 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.185888 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185987 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.187866 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.187869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.187872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.187895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.189429 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.189917 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.189921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.189923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.190022 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.190319 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.190322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.190323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.191077 (+ 754us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.191217 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.194296 (+ 3079us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.194443 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":538,"thread_start_us":88,"threads_started":1}
I20260924 20:20:26.196291 26488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.178709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.18.172.67:39827)
0924 20:20:26.179324 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.179383 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.180488 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.181048 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.181059 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.181713 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:26.182438 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.182477 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.183553 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.183557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.184997 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.185006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.185654 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.185861 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.186661 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:26.186682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:26.189937 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:26.192285 (+ 2348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.192288 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.192290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.192705 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.193053 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.193065 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.193278 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.193767 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:26.193775 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:26.194167 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.194681 (+ 514us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.195088 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":36,"spinlock_wait_cycles":14976,"thread_start_us":702,"threads_started":1}
I20260924 20:20:26.196799 26494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.178709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:33975)
0924 20:20:26.180250 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.180264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.180497 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.181060 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.181063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.182640 (+ 1577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:26.182753 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.182760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.183837 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.183840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.185000 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.185011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.185769 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.185867 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.187085 (+ 1218us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:26.187103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:26.189937 (+ 2834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:26.192185 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.192191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.192208 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.192712 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.193057 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193062 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.193064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.193288 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.193784 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:26.193786 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:26.194166 (+ 380us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.195597 (+ 1431us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.195742 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"mutex_wait_us":125,"spinlock_wait_cycles":47744,"thread_start_us":185,"threads_started":1}
I20260924 20:20:26.196904 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.180311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:54021 (local address 127.18.172.124:33793)
0924 20:20:26.180440 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.180445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.182882 (+ 2437us) server_negotiation.cc:407] Connection header received
0924 20:20:26.183153 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.183156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.183207 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.183255 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.183834 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.184607 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.185740 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.186059 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.186152 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.191157 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.191162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.191166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.191202 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.192996 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193999 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.194002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.194004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.194092 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.194397 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.194400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.194401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.194611 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.194681 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.196634 (+ 1953us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.196767 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:26.197654 26485 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34959
I20260924 20:20:26.197741 26486 heartbeater.cc:347] Connected to a master server at 127.18.172.125:43773
I20260924 20:20:26.197890 26486 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:26.197890 26485 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:26.198364 26491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.179980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.18.172.67:54021)
0924 20:20:26.182789 (+ 2809us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.182802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.183037 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.183374 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.183376 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.183599 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:26.183703 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.183709 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.184764 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.184767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.185619 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.185626 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.186316 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.186318 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.186437 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.187040 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:26.187054 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:26.190964 (+ 3910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:26.193112 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.193114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.193116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.193891 (+ 775us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.194209 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:26.194211 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:26.194213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:26.194288 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:26.194748 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:26.194750 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:26.194826 (+ 76us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.197532 (+ 2706us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.197699 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2749,"thread_start_us":96,"threads_started":1}
I20260924 20:20:26.198532 26485 heartbeater.cc:511] Master 127.18.172.126:34959 requested a full tablet report, sending...
I20260924 20:20:26.198532 26486 heartbeater.cc:511] Master 127.18.172.125:43773 requested a full tablet report, sending...
I20260924 20:20:26.199354 26484 heartbeater.cc:347] Connected to a master server at 127.18.172.124:33793
I20260924 20:20:26.199430 26484 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:26.199630 26484 heartbeater.cc:511] Master 127.18.172.124:33793 requested a full tablet report, sending...
I20260924 20:20:26.199745 25849 ts_manager.cc:194] Registered new tserver with Master: 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:26.200307 25249 ts_manager.cc:194] Re-registered known tserver with Master: 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:26.200479 26035 ts_manager.cc:194] Registered new tserver with Master: 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:26.201087 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:33975
I20260924 20:20:26.203926 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.182243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.18.172.124:33793)
0924 20:20:26.182491 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.182503 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.192152 (+ 9649us) server_negotiation.cc:407] Connection header received
0924 20:20:26.192191 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.192194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.192241 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.192328 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.193218 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.194128 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.197702 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.198047 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.198144 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.199670 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.199675 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.199679 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.199701 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.202000 (+ 2299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.202559 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.202563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.202565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.202655 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.202934 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.202937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.202939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.203200 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.203332 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.203607 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.203730 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":115,"threads_started":1}
I20260924 20:20:26.204180 26497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.181279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.18.172.126:34959)
0924 20:20:26.186750 (+ 5471us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.186755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.186768 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:26.186806 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.186810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.186857 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.188371 (+ 1514us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.188487 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.189357 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.193212 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.193585 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.197152 (+ 3567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.197253 (+ 101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.197255 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.197257 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.197269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.201820 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.202343 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.202347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.202349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.202437 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.202754 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.202758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.202759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.202952 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.203096 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.203889 (+ 793us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.204028 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5388,"thread_start_us":76,"threads_started":1}
I20260924 20:20:26.206634 25249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:26.209770 25249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:26.212214 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: NON_VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: true } } }
I20260924 20:20:26.212355 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:26.215824 25808 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: attempting to promote NON_VOTER df9878cf9f114bf3822e07e1e7b3230d to VOTER
I20260924 20:20:26.216267 25808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:26.217278 25851 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:20:26.217804 26502 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 20:20:26.219219 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 LEARNER]: Refusing update from remote peer 6c03f18dff984dd893e3f7719b2526da: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260924 20:20:26.219412 26502 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.219596 25808 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260924 20:20:26.220070 25851 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.221146 26503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.221256 26503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:26.221146 26045 raft_consensus.cc:2932] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.221652 26502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.221755 26502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:26.222215 26512 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.222375 26512 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:26.222784 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.222877 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:26.223495 25808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6c03f18dff984dd893e3f7719b2526da" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:26.223605 25808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:26.231910 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.226878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:41779 (local address 127.0.0.1:56892)
0924 20:20:26.227045 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.227061 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.227143 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.228144 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.228147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.228164 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.228266 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.228273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.229256 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.229259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.230378 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.230385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.231505 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.231509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.231596 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.231634 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.231709 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.231777 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 20:20:26.231994 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.227154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.18.172.66:41779)
0924 20:20:26.227499 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.227514 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.227537 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:20:26.227650 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.227656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.227910 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.228082 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.228371 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.229119 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.230500 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.231383 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.231491 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.231652 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.231739 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.231840 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":157,"threads_started":1}
I20260924 20:20:26.233042 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.227918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.18.172.67:35541)
0924 20:20:26.228513 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.228522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.228540 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:20:26.228601 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.228604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.228752 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.228917 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.229095 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.230188 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.231699 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.232482 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.232589 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.232718 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.232820 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.232918 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":107,"threads_started":1}
I20260924 20:20:26.234256 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.227190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:43265 (local address 127.0.0.1:59180)
0924 20:20:26.227468 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.227483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.227566 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.230355 (+ 2789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.230358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.230376 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.230480 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.230487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.231707 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.231710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.232806 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.232813 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.233884 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.233888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.233971 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.233993 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.234040 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.234102 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":121,"threads_started":1}
I20260924 20:20:26.234895 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.227695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.18.172.65:43265)
0924 20:20:26.229309 (+ 1614us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.229324 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.229488 (+ 164us) server_negotiation.cc:407] Connection header received
0924 20:20:26.229585 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.230004 (+ 419us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.230205 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.230634 (+ 429us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.230750 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.231579 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.232952 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.233754 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.234429 (+ 675us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.234598 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.234648 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.234738 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":134,"threads_started":1}
I20260924 20:20:26.235369 26282 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 (DEFAULT_TABLE table=test-table [id=624032374a804d409b83857337b15e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:26.236474 26282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbc2159a6a445883c53fa561ba9e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:26.238042 26145 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 (DEFAULT_TABLE table=test-table [id=624032374a804d409b83857337b15e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:26.239177 26145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbc2159a6a445883c53fa561ba9e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:26.243292 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.227533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:35541 (local address 127.0.0.1:59492)
0924 20:20:26.228177 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.228192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.228279 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.228886 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.228889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.228907 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.229002 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.229008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.230298 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.230314 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.231561 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.231571 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.232607 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.232611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.232696 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.232718 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.232794 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.243134 (+ 10340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":83,"threads_started":1}
I20260924 20:20:26.245383 26518 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Bootstrap starting.
I20260924 20:20:26.246348 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 (DEFAULT_TABLE table=test-table [id=624032374a804d409b83857337b15e0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:26.247306 26519 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Bootstrap starting.
I20260924 20:20:26.247387 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dfbc2159a6a445883c53fa561ba9e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:26.247994 26518 tablet_bootstrap.cc:654] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:26.248878 26518 log.cc:824] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:26.249495 26519 tablet_bootstrap.cc:654] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:26.250290 26519 log.cc:824] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:26.250808 26518 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: No bootstrap required, opened a new log
I20260924 20:20:26.251049 26518 ts_tablet_manager.cc:1397] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:20:26.254711 26518 raft_consensus.cc:348] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.254946 26518 raft_consensus.cc:374] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:26.255007 26518 raft_consensus.cc:729] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d7e03495e748f28e6c45ab824b6c98, State: Initialized, Role: FOLLOWER
I20260924 20:20:26.255081 26522 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Bootstrap starting.
I20260924 20:20:26.255533 26518 consensus_queue.cc:260] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.256253 26349 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
I20260924 20:20:26.256518 26518 ts_tablet_manager.cc:1428] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260924 20:20:26.257212 26519 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: No bootstrap required, opened a new log
I20260924 20:20:26.257426 26519 ts_tablet_manager.cc:1397] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260924 20:20:26.262488 26522 tablet_bootstrap.cc:654] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:26.262497 26519 raft_consensus.cc:348] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.262794 26519 raft_consensus.cc:374] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:26.262872 26519 raft_consensus.cc:729] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb843e144828415e8953c52fb2213bb5, State: Initialized, Role: FOLLOWER
I20260924 20:20:26.263370 26522 log.cc:824] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:26.263375 26519 consensus_queue.cc:260] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.264421 26519 ts_tablet_manager.cc:1428] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:20:26.264734 26212 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
I20260924 20:20:26.265405 26522 tablet_bootstrap.cc:492] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: No bootstrap required, opened a new log
I20260924 20:20:26.265610 26522 ts_tablet_manager.cc:1397] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260924 20:20:26.269086 26522 raft_consensus.cc:348] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.269333 26522 raft_consensus.cc:374] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:26.269393 26522 raft_consensus.cc:729] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edad0b4304c42998162c48e774d2a0d, State: Initialized, Role: FOLLOWER
I20260924 20:20:26.269876 26522 consensus_queue.cc:260] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.270851 26522 ts_tablet_manager.cc:1428] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:20:26.271307 26485 heartbeater.cc:503] Master 127.18.172.126:34959 was elected leader, sending a full tablet report...
I20260924 20:20:26.394114 26523 raft_consensus.cc:482] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:26.394325 26523 raft_consensus.cc:504] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.395566 26523 leader_election.cc:288] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265), 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:26.399294 26516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.395935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:36463 (local address 127.18.172.65:43265)
0924 20:20:26.396090 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.396098 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.396120 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:20:26.396208 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.396211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.396262 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.396357 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.396658 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.397174 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398295 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.398701 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398832 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.398886 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.399099 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.399174 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:20:26.399310 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.395846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:43265 (local address 127.18.172.66:36463)
0924 20:20:26.395977 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.395995 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.396105 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.396384 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.396387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.396405 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.396539 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.396546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.397292 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.397298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.398173 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.398182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.398821 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.398930 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.398948 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.399082 (+ 134us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.399162 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:20:26.399420 26517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.396263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52683 (local address 127.18.172.67:35541)
0924 20:20:26.396441 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.396447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.396463 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:26.396515 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.396520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.396569 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.396657 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:26.396913 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.397414 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398437 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.398831 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398961 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.399009 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.399228 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.399295 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:20:26.399425 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.396148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:35541 (local address 127.18.172.66:52683)
0924 20:20:26.396285 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:26.396300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:26.396367 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:26.396666 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:26.396669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:26.396684 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:26.396797 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.396802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.397549 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.397553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.398308 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:26.398316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.398955 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.398959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:26.399104 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.399124 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:20:26.399212 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:20:26.399279 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:20:26.400111 26165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbc2159a6a445883c53fa561ba9e84" candidate_uuid: "d5d7e03495e748f28e6c45ab824b6c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb843e144828415e8953c52fb2213bb5" is_pre_election: true
I20260924 20:20:26.400327 26438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbc2159a6a445883c53fa561ba9e84" candidate_uuid: "d5d7e03495e748f28e6c45ab824b6c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edad0b4304c42998162c48e774d2a0d" is_pre_election: true
I20260924 20:20:26.400454 26165 raft_consensus.cc:2454] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d7e03495e748f28e6c45ab824b6c98 in term 0.
I20260924 20:20:26.400621 26438 raft_consensus.cc:2454] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5d7e03495e748f28e6c45ab824b6c98 in term 0.
I20260924 20:20:26.401022 26237 leader_election.cc:302] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5d7e03495e748f28e6c45ab824b6c98, fb843e144828415e8953c52fb2213bb5; no voters:
I20260924 20:20:26.401328 26523 raft_consensus.cc:2781] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:26.401438 26523 raft_consensus.cc:482] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:26.401544 26523 raft_consensus.cc:3037] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:26.402863 26523 raft_consensus.cc:504] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.403219 26523 leader_election.cc:288] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [CANDIDATE]: Term 1 election: Requested vote from peers fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265), 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:26.403597 26165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbc2159a6a445883c53fa561ba9e84" candidate_uuid: "d5d7e03495e748f28e6c45ab824b6c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb843e144828415e8953c52fb2213bb5"
I20260924 20:20:26.403628 26438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dfbc2159a6a445883c53fa561ba9e84" candidate_uuid: "d5d7e03495e748f28e6c45ab824b6c98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edad0b4304c42998162c48e774d2a0d"
I20260924 20:20:26.403715 26165 raft_consensus.cc:3037] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:26.403759 26438 raft_consensus.cc:3037] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:26.404695 26165 raft_consensus.cc:2454] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d7e03495e748f28e6c45ab824b6c98 in term 1.
I20260924 20:20:26.404805 26438 raft_consensus.cc:2454] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d7e03495e748f28e6c45ab824b6c98 in term 1.
I20260924 20:20:26.404985 26237 leader_election.cc:302] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5d7e03495e748f28e6c45ab824b6c98, fb843e144828415e8953c52fb2213bb5; no voters:
I20260924 20:20:26.405135 26523 raft_consensus.cc:2781] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:26.405359 26523 raft_consensus.cc:686] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 LEADER]: Becoming Leader. State: Replica: d5d7e03495e748f28e6c45ab824b6c98, State: Running, Role: LEADER
I20260924 20:20:26.405738 26523 consensus_queue.cc:237] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:26.409793 25248 catalog_manager.cc:5495] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "d5d7e03495e748f28e6c45ab824b6c98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } health_report { overall_health: UNKNOWN } } }
W20260924 20:20:26.428160 26487 tablet.cc:2369] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:20:26.461490 26351 tablet.cc:2369] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:26.465289 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.461074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56908 (local address 127.18.172.66:41779)
0924 20:20:26.461219 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.461226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.461245 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:26.461468 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.461471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.461529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.461654 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:26.461949 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.462584 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.463675 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.463928 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.464052 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.464407 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:26.464853 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:26.465000 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.465070 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.465132 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:26.471318 26438 raft_consensus.cc:1261] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Refusing update from remote peer d5d7e03495e748f28e6c45ab824b6c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:26.471321 26165 raft_consensus.cc:1261] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Refusing update from remote peer d5d7e03495e748f28e6c45ab824b6c98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:26.472054 26527 consensus_queue.cc:1035] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:26.472280 26523 consensus_queue.cc:1035] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:26.485486 26531 mvcc.cc:205] Tried to move back new op lower bound from 7332991903618600960 to 7332991903360708608. Current Snapshot: MvccSnapshot[applied={T|T < 7332991903618600960}]
I20260924 20:20:26.490455 25248 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:26.490654 25248 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:26.495095 25248 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c03f18dff984dd893e3f7719b2526da: Sending DeleteTablet for 3 replicas of tablet 1dfbc2159a6a445883c53fa561ba9e84
I20260924 20:20:26.496045 26145 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:26 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59180
I20260924 20:20:26.496102 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:26 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59492
I20260924 20:20:26.496215 26282 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dfbc2159a6a445883c53fa561ba9e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:26 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:56892
I20260924 20:20:26.496603 26542 tablet_replica.cc:330] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: stopping tablet replica
I20260924 20:20:26.496927 26542 raft_consensus.cc:2229] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:26.497259 26540 tablet_replica.cc:330] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: stopping tablet replica
I20260924 20:20:26.497359 26542 raft_consensus.cc:2258] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:26.497571 26540 raft_consensus.cc:2229] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:26.497860 26540 raft_consensus.cc:2258] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:26.498942 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25217
I20260924 20:20:26.500727 26540 ts_tablet_manager.cc:1905] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:26.504215 26540 ts_tablet_manager.cc:1918] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:26.504333 26540 log.cc:1196] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/1dfbc2159a6a445883c53fa561ba9e84
I20260924 20:20:26.504662 26540 ts_tablet_manager.cc:1939] T 1dfbc2159a6a445883c53fa561ba9e84 P fb843e144828415e8953c52fb2213bb5: Deleting consensus metadata
I20260924 20:20:26.504693 26542 ts_tablet_manager.cc:1905] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:26.507599 26542 ts_tablet_manager.cc:1918] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:26.507772 26542 log.cc:1196] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/1dfbc2159a6a445883c53fa561ba9e84
W20260924 20:20:26.507993 26098 connection.cc:535] server connection from 127.0.0.1:59180 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:20:26.508257 26542 ts_tablet_manager.cc:1939] T 1dfbc2159a6a445883c53fa561ba9e84 P d5d7e03495e748f28e6c45ab824b6c98: Deleting consensus metadata
W20260924 20:20:26.509310 26235 connection.cc:441] server connection from 127.0.0.1:56892 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56892 (request call id 1) could send its response
I20260924 20:20:26.509665 26541 tablet_replica.cc:330] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: stopping tablet replica
I20260924 20:20:26.510021 26541 raft_consensus.cc:2229] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:26.510455 26541 raft_consensus.cc:2258] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:26.512952 26541 ts_tablet_manager.cc:1905] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:26.515636 26541 ts_tablet_manager.cc:1918] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:26.515738 26541 log.cc:1196] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/1dfbc2159a6a445883c53fa561ba9e84
I20260924 20:20:26.516067 26541 ts_tablet_manager.cc:1939] T 1dfbc2159a6a445883c53fa561ba9e84 P 8edad0b4304c42998162c48e774d2a0d: Deleting consensus metadata
W20260924 20:20:26.517500 26373 connection.cc:441] server connection from 127.0.0.1:59492 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59492 (request call id 1) could send its response
I20260924 20:20:26.520781 26218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.511282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.18.172.124:33793)
0924 20:20:26.511410 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.511416 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.511577 (+ 161us) server_negotiation.cc:407] Connection header received
0924 20:20:26.511764 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.511767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.511819 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.511904 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.512861 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.513518 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.514350 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.514583 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.514709 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.516270 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.516274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.516277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.516304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.518267 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.518881 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.518886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.518890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.518994 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.519572 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.519578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.519582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.519851 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.519969 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.520422 (+ 453us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.520561 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:26.521896 26496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.498527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60496 (local address 127.18.172.125:43773)
0924 20:20:26.498640 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.498644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.498657 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:26.498693 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.498698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.498744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.498815 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.507057 (+ 8242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.507863 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.509625 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.509952 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.510031 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.517721 (+ 7690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.517728 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.517732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.517792 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.519900 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.520380 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.520384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.520386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.520466 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.520753 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.520757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.520758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.521012 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.521121 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.521611 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.521733 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:20:26.529723 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.519221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.18.172.125:43773)
0924 20:20:26.519341 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.519347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.519778 (+ 431us) server_negotiation.cc:407] Connection header received
0924 20:20:26.519970 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.519974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.520031 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.520103 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.521063 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.521837 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.522994 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.523313 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.523378 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.525737 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.525741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.525744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.525804 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.527842 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.528318 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.528325 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.528330 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.528433 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.528816 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.528820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.528822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.529035 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.529163 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.529459 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.529570 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 20:20:26.531663 26499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:26.498236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.18.172.124:33793)
0924 20:20:26.498338 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:26.498344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:26.522855 (+ 24511us) server_negotiation.cc:407] Connection header received
0924 20:20:26.523049 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:26.523052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:26.523098 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:26.523148 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:26.524176 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.524733 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.525826 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:26.526189 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:26.526324 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:26.527805 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:26.527809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:26.527812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:26.527863 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:26.529775 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.530359 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.530365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.530366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.530465 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:26.530771 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:26.530775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:26.530777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:26.531060 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:26.531188 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:26.531396 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 20:20:26.531515 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:20:27.483340 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.482592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:57871)
0924 20:20:27.482847 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.482968 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":93,"threads_started":1}
I20260924 20:20:27.483880 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.483414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:33585)
0924 20:20:27.483702 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.483789 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":69,"threads_started":1}
W20260924 20:20:27.483989 26485 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:34959 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
W20260924 20:20:27.484330 26212 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:34959 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:27.485574 26552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.485208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:43731)
0924 20:20:27.485409 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.485474 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":55,"threads_started":1}
W20260924 20:20:27.485810 26349 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:34959 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:27.995597 26553 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c03f18dff984dd893e3f7719b2526da)
I20260924 20:20:27.995888 26553 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:27.995936 26554 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c03f18dff984dd893e3f7719b2526da)
I20260924 20:20:27.996115 26554 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:27.996954 26553 leader_election.cc:288] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793)
I20260924 20:20:27.996991 26554 leader_election.cc:288] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), b7700bf492734b54b013d5d105ecddea (127.18.172.125:43773)
I20260924 20:20:27.997622 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57416)
0924 20:20:27.997422 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.997516 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":69,"threads_started":1}
W20260924 20:20:27.997794 26017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:27.997932 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57432)
0924 20:20:27.997621 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.997799 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":97,"threads_started":1}
W20260924 20:20:27.998170 25834 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:27.998698 26555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.998458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57436)
0924 20:20:27.998587 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.998635 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 20:20:27.998828 26017 leader_election.cc:334] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:27.999172 26557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.998844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57440)
0924 20:20:27.999067 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.999116 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
W20260924 20:20:27.999264 25834 leader_election.cc:334] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:28.002063 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:43773 (local address 127.0.0.1:60510)
0924 20:20:27.997528 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.997544 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:27.997642 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:27.998102 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:27.998106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:27.998124 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:27.998245 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:27.998252 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:27.999405 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:27.999408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:28.000656 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:28.000666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:28.001551 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.001555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:28.001732 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:28.001755 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:28.001837 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:20:28.001920 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":22,"thread_start_us":103,"threads_started":1}
I20260924 20:20:28.002096 26560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.18.172.124:33793)
0924 20:20:27.998344 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:27.998350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:27.998362 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:27.998407 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:27.998410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:27.998462 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:27.998540 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:27.998892 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:27.999568 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.000989 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:28.001471 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.001600 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:28.001737 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:28.001847 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 20:20:28.001979 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":412,"thread_start_us":62,"threads_started":1}
I20260924 20:20:28.002099 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:33793 (local address 127.0.0.1:35938)
0924 20:20:27.998081 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:27.998096 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:27.998184 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:27.998650 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:27.998653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:27.998670 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:27.998767 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:27.998773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:27.999673 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:27.999676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:28.000838 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:28.000845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:28.001599 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.001603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:28.001705 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:28.001731 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:20:28.001812 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:20:28.001913 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"mutex_wait_us":3,"thread_start_us":55,"threads_started":1}
I20260924 20:20:28.002125 26558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:27.997382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.18.172.125:43773)
0924 20:20:27.997833 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:27.997837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:27.997850 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:27.997897 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:27.997900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:27.997965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:27.998355 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:27.998459 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:27.999277 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.000806 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:28.001441 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:28.001559 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:28.001609 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:28.001852 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 20:20:28.001963 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":396,"thread_start_us":62,"threads_started":1}
I20260924 20:20:28.002669 25851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df9878cf9f114bf3822e07e1e7b3230d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7700bf492734b54b013d5d105ecddea" is_pre_election: true
I20260924 20:20:28.002813 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7700bf492734b54b013d5d105ecddea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df9878cf9f114bf3822e07e1e7b3230d" is_pre_election: true
I20260924 20:20:28.002894 25851 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df9878cf9f114bf3822e07e1e7b3230d in term 1.
I20260924 20:20:28.003057 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7700bf492734b54b013d5d105ecddea in term 1.
I20260924 20:20:28.003268 26018 leader_election.cc:302] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b7700bf492734b54b013d5d105ecddea, df9878cf9f114bf3822e07e1e7b3230d; no voters: 6c03f18dff984dd893e3f7719b2526da
I20260924 20:20:28.003350 25834 leader_election.cc:302] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b7700bf492734b54b013d5d105ecddea, df9878cf9f114bf3822e07e1e7b3230d; no voters: 6c03f18dff984dd893e3f7719b2526da
I20260924 20:20:28.003513 26554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:20:28.003582 26554 raft_consensus.cc:482] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c03f18dff984dd893e3f7719b2526da)
I20260924 20:20:28.003594 26553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 20:20:28.003620 26554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:20:28.003650 26553 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6c03f18dff984dd893e3f7719b2526da)
I20260924 20:20:28.003691 26553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:20:28.004596 26553 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:28.004603 26554 raft_consensus.cc:504] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:28.004866 26553 leader_election.cc:288] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 election: Requested vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793)
I20260924 20:20:28.004947 26554 leader_election.cc:288] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 election: Requested vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), b7700bf492734b54b013d5d105ecddea (127.18.172.125:43773)
I20260924 20:20:28.005230 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.005028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57444)
0924 20:20:28.005113 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.005165 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 20:20:28.005364 25851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df9878cf9f114bf3822e07e1e7b3230d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b7700bf492734b54b013d5d105ecddea"
I20260924 20:20:28.005355 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7700bf492734b54b013d5d105ecddea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df9878cf9f114bf3822e07e1e7b3230d"
I20260924 20:20:28.005468 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.005059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57456)
0924 20:20:28.005295 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.005345 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260924 20:20:28.005532 25851 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df9878cf9f114bf3822e07e1e7b3230d in current term 2: Already voted for candidate b7700bf492734b54b013d5d105ecddea in this term.
I20260924 20:20:28.005582 26045 raft_consensus.cc:2379] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b7700bf492734b54b013d5d105ecddea in current term 2: Already voted for candidate df9878cf9f114bf3822e07e1e7b3230d in this term.
I20260924 20:20:28.006305 26559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.006095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57470)
0924 20:20:28.006192 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.006244 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260924 20:20:28.006379 26556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.006192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57476)
0924 20:20:28.006272 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.006320 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 20:20:28.006433 25834 leader_election.cc:334] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
W20260924 20:20:28.006472 26017 leader_election.cc:334] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:28.006527 25834 leader_election.cc:302] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b7700bf492734b54b013d5d105ecddea; no voters: 6c03f18dff984dd893e3f7719b2526da, df9878cf9f114bf3822e07e1e7b3230d
I20260924 20:20:28.006551 26017 leader_election.cc:302] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: df9878cf9f114bf3822e07e1e7b3230d; no voters: 6c03f18dff984dd893e3f7719b2526da, b7700bf492734b54b013d5d105ecddea
I20260924 20:20:28.006659 26553 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:20:28.006793 26554 raft_consensus.cc:2726] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 20:20:28.485646 26563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.484639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:32907)
0924 20:20:28.485456 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.485532 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":99,"threads_started":1}
I20260924 20:20:28.485725 26564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.485249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:56715)
0924 20:20:28.485589 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.485636 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":74,"threads_started":1}
I20260924 20:20:28.486716 26565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:28.486307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:57573)
0924 20:20:28.486535 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:28.486600 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":71,"threads_started":1}
I20260924 20:20:29.487288 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.486394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:40601)
0924 20:20:29.486744 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.487156 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":80,"threads_started":1}
I20260924 20:20:29.487468 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.486774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:42567)
0924 20:20:29.487279 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.487348 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":77,"threads_started":1}
I20260924 20:20:29.488157 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.487336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:55419)
0924 20:20:29.487986 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.488054 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":85,"threads_started":1}
I20260924 20:20:29.727512 26569 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:29.727654 26569 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:29.728179 26569 leader_election.cc:288] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793)
I20260924 20:20:29.729043 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.728569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57708)
0924 20:20:29.728836 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.728948 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
I20260924 20:20:29.729261 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7700bf492734b54b013d5d105ecddea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df9878cf9f114bf3822e07e1e7b3230d" is_pre_election: true
I20260924 20:20:29.729430 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b7700bf492734b54b013d5d105ecddea in term 2.
I20260924 20:20:29.729871 25834 leader_election.cc:302] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7700bf492734b54b013d5d105ecddea, df9878cf9f114bf3822e07e1e7b3230d; no voters:
I20260924 20:20:29.730055 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.729709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57718)
0924 20:20:29.729897 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.729976 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 20:20:29.730113 26569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 20:20:29.730163 26569 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260924 20:20:29.730175 25834 leader_election.cc:334] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:29.730217 26569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:20:29.731372 26569 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:29.731709 26569 leader_election.cc:288] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 election: Requested vote from peers 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959), df9878cf9f114bf3822e07e1e7b3230d (127.18.172.124:33793)
I20260924 20:20:29.732072 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.731819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57732)
0924 20:20:29.731931 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.732007 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 20:20:29.732257 26045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7700bf492734b54b013d5d105ecddea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "df9878cf9f114bf3822e07e1e7b3230d"
I20260924 20:20:29.732502 26045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:20:29.733594 26045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7700bf492734b54b013d5d105ecddea in term 3.
I20260924 20:20:29.733930 25834 leader_election.cc:302] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b7700bf492734b54b013d5d105ecddea, df9878cf9f114bf3822e07e1e7b3230d; no voters:
I20260924 20:20:29.734160 26569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:20:29.734376 26569 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [term 3 LEADER]: Becoming Leader. State: Replica: b7700bf492734b54b013d5d105ecddea, State: Running, Role: LEADER
I20260924 20:20:29.734478 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.734236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57748)
0924 20:20:29.734374 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.734423 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260924 20:20:29.734556 25834 leader_election.cc:334] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
I20260924 20:20:29.734673 26569 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } }
I20260924 20:20:29.735893 26572 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7700bf492734b54b013d5d105ecddea. Latest consensus state: current_term: 3 leader_uuid: "b7700bf492734b54b013d5d105ecddea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:29.735999 26572 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:29.736290 26574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:29.738596 26574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:29.738991 26574 catalog_manager.cc:1259] Loaded cluster ID: ff1caf86e16841febce96b6acb2c6866
I20260924 20:20:29.739087 26574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:29.739650 26574 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:29.740032 26574 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Loaded TSK: 0
I20260924 20:20:29.740748 26574 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:29.760927 25849 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:29.764190 25849 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:29.767323 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.767076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57762)
0924 20:20:29.767201 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.767257 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 20:20:29.767282 26045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [term 3 FOLLOWER]: Refusing update from remote peer b7700bf492734b54b013d5d105ecddea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 20:20:29.767833 26569 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [LEADER]: Connected to new peer: Peer: permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 20:20:29.768133 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.767918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.0.0.1:57778)
0924 20:20:29.768031 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.768083 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:20:29.768319 25834 consensus_peers.cc:487] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea -> Peer 6c03f18dff984dd893e3f7719b2526da (127.18.172.126:34959): Couldn't send request to peer 6c03f18dff984dd893e3f7719b2526da. Status: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 20:20:29.769275 26576 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7700bf492734b54b013d5d105ecddea. Latest consensus state: current_term: 3 leader_uuid: "b7700bf492734b54b013d5d105ecddea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:29.769510 26576 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:29.770512 26569 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b7700bf492734b54b013d5d105ecddea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:29.770648 26569 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:29.771040 26576 sys_catalog.cc:455] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b7700bf492734b54b013d5d105ecddea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "6c03f18dff984dd893e3f7719b2526da" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34959 } } peers { permanent_uuid: "b7700bf492734b54b013d5d105ecddea" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 43773 } attrs { promote: false } } peers { permanent_uuid: "df9878cf9f114bf3822e07e1e7b3230d" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 33793 } attrs { promote: false } } }
I20260924 20:20:29.771152 26576 sys_catalog.cc:458] T 00000000000000000000000000000000 P df9878cf9f114bf3822e07e1e7b3230d [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:20:29.771211 26577 mvcc.cc:205] Tried to move back new op lower bound from 7332991917121626112 to 7332991916996390912. Current Snapshot: MvccSnapshot[applied={T|T < 7332991917121626112 or (T in {7332991917121626112})}]
I20260924 20:20:29.780908 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.776230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:41779 (local address 127.0.0.1:56922)
0924 20:20:29.776656 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.776670 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:29.776769 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:29.777035 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:29.777040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:29.777058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:29.777186 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.777195 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.778352 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.778355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.779632 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.779643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.780505 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.780508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.780611 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.780647 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 20:20:29.780695 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 20:20:29.780773 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":42,"spinlock_wait_cycles":1280}
I20260924 20:20:29.781126 26582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.776757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:35541 (local address 127.0.0.1:59494)
0924 20:20:29.777135 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.777147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:29.777222 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:29.777595 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:29.777598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:29.777615 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:29.777687 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.777692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.778707 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.778710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.779727 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.779739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.780706 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.780709 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.780807 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.780828 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:20:29.780894 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:20:29.780967 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":24,"thread_start_us":103,"threads_started":1}
I20260924 20:20:29.781445 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.776354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56922 (local address 127.18.172.66:41779)
0924 20:20:29.776604 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.776610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.776770 (+ 160us) server_negotiation.cc:407] Connection header received
0924 20:20:29.776866 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.776869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.776921 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.777311 (+ 390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:29.777413 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.778230 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.779760 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.780376 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.781120 (+ 744us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.781195 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.781230 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.781299 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
I20260924 20:20:29.781473 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.777108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59494 (local address 127.18.172.67:35541)
0924 20:20:29.777339 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.777345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.777361 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:29.777415 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.777417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.777480 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.777569 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:29.777834 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.778590 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.779855 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.780494 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.780590 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.780653 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.781262 (+ 609us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.781348 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":79,"threads_started":1}
I20260924 20:20:29.781472 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.776436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:43265 (local address 127.0.0.1:59188)
0924 20:20:29.776979 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.776992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:29.777083 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:29.778190 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:29.778193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:29.778208 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:29.778306 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.778312 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.779068 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.779071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.780254 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.780263 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.781119 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.781123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.781217 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.781247 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:20:29.781295 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 20:20:29.781347 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":24,"thread_start_us":95,"threads_started":1}
I20260924 20:20:29.781934 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.776611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.18.172.65:43265)
0924 20:20:29.777917 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.777975 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.777989 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:29.778045 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.778047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.778086 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.778182 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:29.778426 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.778917 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.780387 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.780996 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.781637 (+ 641us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.781697 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.781732 (+ 35us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.781799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1226,"thread_start_us":96,"threads_started":1}
I20260924 20:20:29.781929 26282 tablet_service.cc:1467] Processing CreateTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 (DEFAULT_TABLE table=test-table [id=f0736de8647a4e6eb9b602930579d31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:29.781937 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 (DEFAULT_TABLE table=test-table [id=f0736de8647a4e6eb9b602930579d31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:29.782228 26282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56110272bf3c48d5b1eedde8e84a05c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:29.782229 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56110272bf3c48d5b1eedde8e84a05c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:29.782429 26145 tablet_service.cc:1467] Processing CreateTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 (DEFAULT_TABLE table=test-table [id=f0736de8647a4e6eb9b602930579d31c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:29.782680 26145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56110272bf3c48d5b1eedde8e84a05c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:29.784457 26586 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Bootstrap starting.
I20260924 20:20:29.785116 26588 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: Bootstrap starting.
I20260924 20:20:29.785380 26586 tablet_bootstrap.cc:654] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:29.785885 26587 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Bootstrap starting.
I20260924 20:20:29.786026 26588 tablet_bootstrap.cc:654] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:29.786669 26586 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: No bootstrap required, opened a new log
I20260924 20:20:29.786777 26586 ts_tablet_manager.cc:1397] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:29.786868 26587 tablet_bootstrap.cc:654] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:29.787351 26588 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: No bootstrap required, opened a new log
I20260924 20:20:29.787281 26586 raft_consensus.cc:348] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.787401 26586 raft_consensus.cc:374] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:29.787441 26588 ts_tablet_manager.cc:1397] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:29.787454 26586 raft_consensus.cc:729] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edad0b4304c42998162c48e774d2a0d, State: Initialized, Role: FOLLOWER
I20260924 20:20:29.787590 26586 consensus_queue.cc:260] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.787947 26586 ts_tablet_manager.cc:1428] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260924 20:20:29.787887 26588 raft_consensus.cc:348] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.788004 26587 tablet_bootstrap.cc:492] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: No bootstrap required, opened a new log
I20260924 20:20:29.788003 26588 raft_consensus.cc:374] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:29.788040 26588 raft_consensus.cc:729] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb843e144828415e8953c52fb2213bb5, State: Initialized, Role: FOLLOWER
I20260924 20:20:29.788079 26587 ts_tablet_manager.cc:1397] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:29.788185 26588 consensus_queue.cc:260] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.788580 26588 ts_tablet_manager.cc:1428] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:29.788761 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.788361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:51643)
0924 20:20:29.788630 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.788686 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260924 20:20:29.789718 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.789471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:55195)
0924 20:20:29.789595 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.789653 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 20:20:29.790081 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.789824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:58473)
0924 20:20:29.789949 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.790004 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 20:20:29.791257 26486 heartbeater.cc:503] Master 127.18.172.125:43773 was elected leader, sending a full tablet report...
I20260924 20:20:29.791934 26587 raft_consensus.cc:348] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.792060 26587 raft_consensus.cc:374] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:29.792102 26587 raft_consensus.cc:729] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d7e03495e748f28e6c45ab824b6c98, State: Initialized, Role: FOLLOWER
I20260924 20:20:29.792241 26587 consensus_queue.cc:260] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.792654 26587 ts_tablet_manager.cc:1428] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260924 20:20:29.793155 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.792908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:44147)
0924 20:20:29.793021 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.793077 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 20:20:29.793853 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.793632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:42323)
0924 20:20:29.793744 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.793794 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:20:29.793962 26350 heartbeater.cc:503] Master 127.18.172.125:43773 was elected leader, sending a full tablet report...
I20260924 20:20:29.931383 26593 raft_consensus.cc:482] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:29.931535 26593 raft_consensus.cc:504] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.932685 26593 leader_election.cc:288] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779), 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:29.936632 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.933084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:35245 (local address 127.18.172.66:41779)
0924 20:20:29.933237 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.933242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.933257 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:20:29.933386 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.933395 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.933507 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.933592 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:29.933877 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.934458 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.935677 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.936069 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.936211 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.936265 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.936450 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.936521 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:29.936632 26566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.932966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:41779 (local address 127.18.172.65:35245)
0924 20:20:29.933118 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.933135 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:29.933256 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:29.933614 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:29.933618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:29.933638 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:29.933767 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.933772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.934618 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.934621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.935549 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.935558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.936212 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.936215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.936311 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.936337 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:20:29.936423 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:20:29.936495 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 20:20:29.936987 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.932966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:35541 (local address 127.18.172.65:42255)
0924 20:20:29.933579 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.933596 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:29.933718 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:29.933964 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:29.933967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:29.933982 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:29.934095 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.934104 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.934920 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.934926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.935822 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:29.935832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.936560 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.936563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:29.936667 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.936690 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:29.936765 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:20:29.936855 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":312,"thread_start_us":149,"threads_started":1}
I20260924 20:20:29.937192 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56110272bf3c48d5b1eedde8e84a05c0" candidate_uuid: "fb843e144828415e8953c52fb2213bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d7e03495e748f28e6c45ab824b6c98" is_pre_election: true
I20260924 20:20:29.937459 26585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.933091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:42255 (local address 127.18.172.67:35541)
0924 20:20:29.933221 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.933226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.933678 (+ 452us) server_negotiation.cc:407] Connection header received
0924 20:20:29.933827 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.933830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.933871 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.933955 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:29.934233 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.934815 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.935956 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.936428 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.936571 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.936624 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.937236 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.937310 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:20:29.937448 26302 raft_consensus.cc:2454] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb843e144828415e8953c52fb2213bb5 in term 0.
I20260924 20:20:29.937822 26438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56110272bf3c48d5b1eedde8e84a05c0" candidate_uuid: "fb843e144828415e8953c52fb2213bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edad0b4304c42998162c48e774d2a0d" is_pre_election: true
I20260924 20:20:29.937882 26099 leader_election.cc:302] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5d7e03495e748f28e6c45ab824b6c98, fb843e144828415e8953c52fb2213bb5; no voters:
I20260924 20:20:29.937973 26438 raft_consensus.cc:2454] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb843e144828415e8953c52fb2213bb5 in term 0.
I20260924 20:20:29.938189 26593 raft_consensus.cc:2781] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:29.938262 26593 raft_consensus.cc:482] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:29.938304 26593 raft_consensus.cc:3037] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:29.939522 26593 raft_consensus.cc:504] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.939843 26593 leader_election.cc:288] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [CANDIDATE]: Term 1 election: Requested vote from peers d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779), 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541)
I20260924 20:20:29.940161 26302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56110272bf3c48d5b1eedde8e84a05c0" candidate_uuid: "fb843e144828415e8953c52fb2213bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5d7e03495e748f28e6c45ab824b6c98"
I20260924 20:20:29.940176 26438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56110272bf3c48d5b1eedde8e84a05c0" candidate_uuid: "fb843e144828415e8953c52fb2213bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8edad0b4304c42998162c48e774d2a0d"
I20260924 20:20:29.940300 26438 raft_consensus.cc:3037] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:29.940310 26302 raft_consensus.cc:3037] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:29.941185 26438 raft_consensus.cc:2454] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb843e144828415e8953c52fb2213bb5 in term 1.
I20260924 20:20:29.941274 26302 raft_consensus.cc:2454] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb843e144828415e8953c52fb2213bb5 in term 1.
I20260924 20:20:29.941466 26100 leader_election.cc:302] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8edad0b4304c42998162c48e774d2a0d, fb843e144828415e8953c52fb2213bb5; no voters:
I20260924 20:20:29.941638 26593 raft_consensus.cc:2781] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:29.941850 26593 raft_consensus.cc:686] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 1 LEADER]: Becoming Leader. State: Replica: fb843e144828415e8953c52fb2213bb5, State: Running, Role: LEADER
I20260924 20:20:29.942086 26213 heartbeater.cc:503] Master 127.18.172.125:43773 was elected leader, sending a full tablet report...
I20260924 20:20:29.942116 26593 consensus_queue.cc:237] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } }
I20260924 20:20:29.942772 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.942504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:51821)
0924 20:20:29.942648 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.942707 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 20:20:29.944905 25849 catalog_manager.cc:5495] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb843e144828415e8953c52fb2213bb5 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "fb843e144828415e8953c52fb2213bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb843e144828415e8953c52fb2213bb5" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 43265 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:29.975322 26583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.971762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.18.172.65:43265)
0924 20:20:29.971907 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:29.971914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:29.971930 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:29.972104 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:29.972106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:29.972163 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:29.972276 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:29.972518 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.973023 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.973891 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:29.974114 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:29.974225 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:29.974500 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:29.974935 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:29.975092 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:29.975124 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 20:20:29.975179 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:20:29.977989 26438 raft_consensus.cc:1261] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Refusing update from remote peer fb843e144828415e8953c52fb2213bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:29.978369 26302 raft_consensus.cc:1261] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Refusing update from remote peer fb843e144828415e8953c52fb2213bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:29.978569 26596 consensus_queue.cc:1035] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8edad0b4304c42998162c48e774d2a0d" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 35541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:29.978972 26593 consensus_queue.cc:1035] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5d7e03495e748f28e6c45ab824b6c98" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:29.981215 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.980841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:37551)
0924 20:20:29.981061 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.981131 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 20:20:29.981712 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.981104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:53699)
0924 20:20:29.981563 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.981635 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
I20260924 20:20:29.984010 26600 mvcc.cc:205] Tried to move back new op lower bound from 7332991917984993280 to 7332991917845413888. Current Snapshot: MvccSnapshot[applied={T|T < 7332991917984993280}]
I20260924 20:20:29.984906 26568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.984677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.66:56915)
0924 20:20:29.984769 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.984828 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 20:20:29.985523 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.985026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.67:50695)
0924 20:20:29.985416 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.985473 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321}
I20260924 20:20:29.988063 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.987805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:48897)
0924 20:20:29.987919 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.987980 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 20:20:29.989202 26595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:29.988888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34959 (local address 127.18.172.65:48629)
0924 20:20:29.989086 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:29.989138 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:34959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260924 20:20:29.992494 25847 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:29.992624 25847 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:29.996284 25847 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b7700bf492734b54b013d5d105ecddea: Sending DeleteTablet for 3 replicas of tablet 56110272bf3c48d5b1eedde8e84a05c0
I20260924 20:20:29.996966 26282 tablet_service.cc:1514] Processing DeleteTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:29 UTC) from {username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:56922
I20260924 20:20:29.997509 26145 tablet_service.cc:1514] Processing DeleteTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:29 UTC) from {username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:59188
I20260924 20:20:29.997803 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet 56110272bf3c48d5b1eedde8e84a05c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:29 UTC) from {username='oryx', principal='oryx/127.18.172.125@KRBTEST.COM'} at 127.0.0.1:59494
I20260924 20:20:29.997877 26610 tablet_replica.cc:330] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5: stopping tablet replica
I20260924 20:20:29.998092 26610 raft_consensus.cc:2229] T 56110272bf3c48d5b1eedde8e84a05c0 P fb843e144828415e8953c52fb2213bb5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:29.998132 26611 tablet_replica.cc:330] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: stopping tablet replica
I20260924 20:20:29.998147 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25993
I20260924 20:20:29.998279 26611 raft_consensus.cc:2229] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:29.998430 26611 raft_consensus.cc:2258] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:29.998732 26609 tablet_replica.cc:330] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: stopping tablet replica
I20260924 20:20:29.998984 26609 raft_consensus.cc:2229] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:29.999151 26609 raft_consensus.cc:2258] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:30.000195 26609 ts_tablet_manager.cc:1905] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:30.003038 26611 ts_tablet_manager.cc:1905] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:30.003830 26609 ts_tablet_manager.cc:1918] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:30.003916 26609 log.cc:1196] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/56110272bf3c48d5b1eedde8e84a05c0
I20260924 20:20:30.004220 26609 ts_tablet_manager.cc:1939] T 56110272bf3c48d5b1eedde8e84a05c0 P d5d7e03495e748f28e6c45ab824b6c98: Deleting consensus metadata
I20260924 20:20:30.005033 25834 catalog_manager.cc:4841] TS d5d7e03495e748f28e6c45ab824b6c98 (127.18.172.66:41779): tablet 56110272bf3c48d5b1eedde8e84a05c0 (table test-table [id=f0736de8647a4e6eb9b602930579d31c]) successfully deleted
I20260924 20:20:30.005275 26611 ts_tablet_manager.cc:1918] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:30.005370 26611 log.cc:1196] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/56110272bf3c48d5b1eedde8e84a05c0
I20260924 20:20:30.005630 26611 ts_tablet_manager.cc:1939] T 56110272bf3c48d5b1eedde8e84a05c0 P 8edad0b4304c42998162c48e774d2a0d: Deleting consensus metadata
I20260924 20:20:30.006274 25835 catalog_manager.cc:4841] TS 8edad0b4304c42998162c48e774d2a0d (127.18.172.67:35541): tablet 56110272bf3c48d5b1eedde8e84a05c0 (table test-table [id=f0736de8647a4e6eb9b602930579d31c]) successfully deleted
I20260924 20:20:30.008337 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 26219
I20260924 20:20:30.008440 26584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:30.008170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:43265 (local address 127.0.0.1:59206)
0924 20:20:30.008294 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:30.008340 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:20:30.009378 25835 catalog_manager.cc:4584] TS fb843e144828415e8953c52fb2213bb5 (127.18.172.65:43265): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56110272bf3c48d5b1eedde8e84a05c0: Network error: Client connection negotiation failed: client connection to 127.18.172.65:43265: connect: Connection refused (error 111)
I20260924 20:20:30.015601 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 26355
I20260924 20:20:30.021237 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25713
I20260924 20:20:30.027422 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 25821
2026-09-24T20:20:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7265 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:30 dist-test-slave-m1qq krb5kdc[26614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:30 dist-test-slave-m1qq krb5kdc[26614](info): set up 2 sockets
Sep 24 20:20:30 dist-test-slave-m1qq krb5kdc[26614](info): commencing operation
krb5kdc: starting...
W20260924 20:20:31.136821 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:31Z Disabled control of system clock
WARNING: no policy specified for oryx/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab.
Entry for principal oryx/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab.
I20260924 20:20:31.267068 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41343
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab
--principal=oryx/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:31.380771 26627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:31.381062 26627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:31.381143 26627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:31.385255 26627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:31.385322 26627 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 20:20:31.385340 26627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:31.385358 26627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:31.385421 26627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:31.385453 26627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:31.391127 26627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41343
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.126.keytab
--principal=oryx/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:31.392243 26627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:31.393334 26627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:31.402299 26627 server_base.cc:1034] running on GCE node
W20260924 20:20:31.402248 26633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:31.402261 26632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:31.402248 26635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:31.403249 26627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:31.404386 26627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:31.405597 26627 hybrid_clock.cc:648] HybridClock initialized: now 1790281231405538 us; error 72 us; skew 500 ppm
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:31.408828 26627 init.cc:377] Logged in from keytab as oryx/127.18.172.126@KRBTEST.COM (short username oryx)
I20260924 20:20:31.409798 26627 webserver.cc:460] Webserver started at http://127.18.172.126:42911/ using document root <none> and password file <none>
I20260924 20:20:31.410387 26627 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:31.410553 26627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:31.410794 26627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:31.413065 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "8ac9ad9563b54d05b39a817e569363f7"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "7fdd4209a5d4b4e0e798790f454bedf1"
server_key_iv: "21ff43b4aebdb49766a2fef126f6a8dc"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.413658 26627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "8ac9ad9563b54d05b39a817e569363f7"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "7fdd4209a5d4b4e0e798790f454bedf1"
server_key_iv: "21ff43b4aebdb49766a2fef126f6a8dc"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.417534 26627 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:20:31.420109 26642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:31.421394 26627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:31.421582 26627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "8ac9ad9563b54d05b39a817e569363f7"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "7fdd4209a5d4b4e0e798790f454bedf1"
server_key_iv: "21ff43b4aebdb49766a2fef126f6a8dc"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.421697 26627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:31.445037 26627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:31.448134 26627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:31.448350 26627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:31.457077 26627 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41343
I20260924 20:20:31.457083 26704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41343 every 8 connection(s)
I20260924 20:20:31.458086 26627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:31.461012 26705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:31.463713 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 26627
I20260924 20:20:31.463860 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:31.464143 19121 external_mini_cluster.cc:1428] Setting key 55f768238ffe9ecacdb253256f61c7db
I20260924 20:20:31.466985 26705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Bootstrap starting.
I20260924 20:20:31.469576 26705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:31.470441 26705 log.cc:824] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:31.472344 26705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: No bootstrap required, opened a new log
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:31.476554 26705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } }
I20260924 20:20:31.476809 26705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:31.476882 26705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac9ad9563b54d05b39a817e569363f7, State: Initialized, Role: FOLLOWER
I20260924 20:20:31.477381 26705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } }
I20260924 20:20:31.477558 26705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:31.477666 26705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:31.477774 26705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:31.478645 26705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } }
I20260924 20:20:31.478899 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.465504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.18.172.126:41343)
0924 20:20:31.465909 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:31.465926 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:31.465963 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:20:31.466573 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:31.466591 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:31.466922 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:31.467304 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:31.468161 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.469179 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.470380 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.470896 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.471142 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.473925 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:31.473930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:31.473943 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:31.473985 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:31.475729 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.476216 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.476221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.476225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.476355 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.476651 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.476653 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.476655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.477138 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:31.477279 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:31.477706 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 20:20:31.478064 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":139,"threads_started":1}
I20260924 20:20:31.479008 26705 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac9ad9563b54d05b39a817e569363f7; no voters:
I20260924 20:20:31.479368 26705 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:31.479456 26710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:31.479694 26710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac9ad9563b54d05b39a817e569363f7, State: Running, Role: LEADER
I20260924 20:20:31.480086 26710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } }
I20260924 20:20:31.480579 26705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:31.481343 26710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac9ad9563b54d05b39a817e569363f7. Latest consensus state: current_term: 1 leader_uuid: "8ac9ad9563b54d05b39a817e569363f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } } }
I20260924 20:20:31.482131 26710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:31.482007 26711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac9ad9563b54d05b39a817e569363f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac9ad9563b54d05b39a817e569363f7" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41343 } } }
I20260924 20:20:31.483091 26711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7 [sys.catalog]: This master's current role is: LEADER
W20260924 20:20:31.485394 26725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:20:31.485462 26725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:20:31.485540 26724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:31.486233 26724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:31.492211 26724 catalog_manager.cc:1347] Generated new cluster ID: 61cec5a09c1d4b2bbd1026d1e0074d29
I20260924 20:20:31.492293 26724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:31.506660 26724 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:31.507728 26724 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:31.513254 26724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Generated new TSK 0
I20260924 20:20:31.513994 26724 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
I20260924 20:20:31.546093 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41343
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:31.661347 26731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:31.661630 26731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:31.661725 26731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:31.666412 26731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:31.666483 26731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:31.666527 26731 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:31.666625 26731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:31.673780 26731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:31.674975 26731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:31.676164 26731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:31.686736 26739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:31.686794 26737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:31.686756 26731 server_base.cc:1034] running on GCE node
W20260924 20:20:31.686764 26736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:31.687448 26731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:31.688071 26731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:31.689226 26731 hybrid_clock.cc:648] HybridClock initialized: now 1790281231689212 us; error 42 us; skew 500 ppm
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:31.691927 26731 init.cc:377] Logged in from keytab as oryx/127.18.172.65@KRBTEST.COM (short username oryx)
I20260924 20:20:31.692744 26731 webserver.cc:460] Webserver started at http://127.18.172.65:33673/ using document root <none> and password file <none>
I20260924 20:20:31.693249 26731 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:31.693306 26731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:31.693472 26731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:31.695510 26731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "926db20805024cca9997a8d53c964fc2"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "3bb8a78aa6269a082110ade7c292a9aa"
server_key_iv: "192ae038d652cb4fd191c960f949faa1"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.696007 26731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "926db20805024cca9997a8d53c964fc2"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "3bb8a78aa6269a082110ade7c292a9aa"
server_key_iv: "192ae038d652cb4fd191c960f949faa1"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.699730 26731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:20:31.702266 26746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:31.703631 26731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:31.703764 26731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "926db20805024cca9997a8d53c964fc2"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "3bb8a78aa6269a082110ade7c292a9aa"
server_key_iv: "192ae038d652cb4fd191c960f949faa1"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.703879 26731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:31.716255 26731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:31.719677 26731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:31.719897 26731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:31.721359 26731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:31.721410 26731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:31.721488 26731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:31.721519 26731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:31.732648 26731 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46533
I20260924 20:20:31.732710 26861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46533 every 8 connection(s)
I20260924 20:20:31.733989 26731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:31.736346 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.722326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:55283 (local address 127.18.172.126:41343)
0924 20:20:31.722465 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:31.722469 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:31.723631 (+ 1162us) server_negotiation.cc:407] Connection header received
0924 20:20:31.724892 (+ 1261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:31.724897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:31.724977 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:31.725088 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:31.726748 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.727686 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.729009 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.729269 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.729390 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.732143 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:31.732152 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:31.732154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:31.732204 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:31.733884 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.734541 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.734545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.734546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.734642 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.735165 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.735169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.735171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.735415 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:31.735562 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:31.736094 (+ 532us) server_negotiation.cc:299] Negotiation successful
0924 20:20:31.736222 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:20:31.737298 26755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.722904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.65:55283)
0924 20:20:31.723437 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:31.723518 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:31.724593 (+ 1075us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:31.725292 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:31.725304 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:31.725794 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:31.726587 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.726612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.727821 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.727824 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.728874 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.728883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.729429 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.729433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.729625 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.730271 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:31.730287 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:31.731910 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:31.734035 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.734041 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.734057 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.734411 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.734768 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.734772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.734774 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.734976 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.735560 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:31.735566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:31.735820 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:20:31.735990 (+ 170us) client_negotiation.cc:240] Negotiation successful
0924 20:20:31.736311 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":142,"threads_started":1}
I20260924 20:20:31.738833 26862 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41343
I20260924 20:20:31.739051 26862 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:31.739614 26862 heartbeater.cc:511] Master 127.18.172.126:41343 requested a full tablet report, sending...
I20260924 20:20:31.741372 26658 ts_manager.cc:194] Registered new tserver with Master: 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533)
I20260924 20:20:31.742873 26658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:55283
I20260924 20:20:31.743518 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 26731
I20260924 20:20:31.743652 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:31.743999 19121 external_mini_cluster.cc:1428] Setting key 11928da08c0cb0220b3a87cde8b88380
I20260924 20:20:31.748720 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.741069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:60825 (local address 127.18.172.126:41343)
0924 20:20:31.741196 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:31.741201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:31.741374 (+ 173us) server_negotiation.cc:407] Connection header received
0924 20:20:31.741475 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:31.741478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:31.741523 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:31.741603 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:31.741997 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.742772 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.743834 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.744164 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.744282 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.745599 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:31.745602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:31.745604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:31.745623 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:31.747112 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.747602 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.747605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.747606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.747675 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.747964 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.747966 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.747967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.748114 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:31.748187 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:31.748528 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 20:20:31.748613 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:31.748911 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.741017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.65:60825)
0924 20:20:31.741285 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:31.741300 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:31.741387 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:31.741619 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:31.741621 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:31.741788 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:31.741886 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.741891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.742915 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.742921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.743714 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.743723 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.744300 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.744303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.744394 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.744962 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:31.744983 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:31.745402 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:31.747218 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.747223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.747226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.747502 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.747763 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.747766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.747768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.747867 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.748208 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:31.748214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:31.748335 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 20:20:31.748457 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 20:20:31.748618 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for oryx/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
I20260924 20:20:31.774703 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41343
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:31.886524 26871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:31.886766 26871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:31.886811 26871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:31.890856 26871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:31.890918 26871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:31.890956 26871 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:31.890990 26871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:31.896276 26871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:31.897320 26871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:31.898299 26871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:31.908212 26879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:31.908229 26876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:31.908259 26877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:31.908558 26871 server_base.cc:1034] running on GCE node
I20260924 20:20:31.909049 26871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:31.909693 26871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:31.910858 26871 hybrid_clock.cc:648] HybridClock initialized: now 1790281231910834 us; error 35 us; skew 500 ppm
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:31.913647 26871 init.cc:377] Logged in from keytab as oryx/127.18.172.66@KRBTEST.COM (short username oryx)
I20260924 20:20:31.914692 26871 webserver.cc:460] Webserver started at http://127.18.172.66:33141/ using document root <none> and password file <none>
I20260924 20:20:31.915405 26871 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:31.915500 26871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:31.915738 26871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:31.917892 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "ce5666e3d55d480ab373d1e68e794024"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "35ba63fd6eea1493303e0540a2038344"
server_key_iv: "81f40b472cc568fabba4fbc61296c2a4"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.918479 26871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "ce5666e3d55d480ab373d1e68e794024"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "35ba63fd6eea1493303e0540a2038344"
server_key_iv: "81f40b472cc568fabba4fbc61296c2a4"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.922427 26871 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:31.925238 26886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:31.926623 26871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:31.926723 26871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "ce5666e3d55d480ab373d1e68e794024"
format_stamp: "Formatted at 2026-09-24 20:20:31 on dist-test-slave-m1qq"
server_key: "35ba63fd6eea1493303e0540a2038344"
server_key_iv: "81f40b472cc568fabba4fbc61296c2a4"
server_key_version: "encryptionkey@0"
I20260924 20:20:31.926815 26871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:31.945196 26871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:31.948351 26871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:31.948522 26871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:31.949993 26871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:31.950049 26871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:31.950090 26871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:31.950146 26871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:31 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:31.961820 26871 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:45419
I20260924 20:20:31.961846 27001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:45419 every 8 connection(s)
I20260924 20:20:31.962803 26871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:31.964808 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.950775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:33523 (local address 127.18.172.126:41343)
0924 20:20:31.950918 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:31.950950 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:31.952106 (+ 1156us) server_negotiation.cc:407] Connection header received
0924 20:20:31.953223 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:31.953228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:31.953292 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:31.953362 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:31.954846 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.955534 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.956805 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.957032 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.957151 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.960844 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:31.960848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:31.960851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:31.960883 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:31.962476 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.963092 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.963100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.963101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.963200 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.963689 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.963692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.963694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.963915 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:31.964018 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:31.964517 (+ 499us) server_negotiation.cc:299] Negotiation successful
0924 20:20:31.964640 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:20:31.965629 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.951386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.66:33523)
0924 20:20:31.951912 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:31.951977 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:31.952949 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:31.953824 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:31.953832 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:31.954168 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:31.954638 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.954676 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.955665 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.955669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.956673 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.956682 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.957206 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.957210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.957395 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.958121 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:31.958148 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:31.960652 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:31.962597 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.962603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.962617 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.962969 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.963340 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.963346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.963349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.963577 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.964025 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:31.964034 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:31.964274 (+ 240us) client_negotiation.cc:769] Sending connection context
0924 20:20:31.964457 (+ 183us) client_negotiation.cc:240] Negotiation successful
0924 20:20:31.964771 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":144,"threads_started":1}
I20260924 20:20:31.966912 27002 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41343
I20260924 20:20:31.967103 27002 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:31.967651 27002 heartbeater.cc:511] Master 127.18.172.126:41343 requested a full tablet report, sending...
I20260924 20:20:31.968673 26658 ts_manager.cc:194] Registered new tserver with Master: ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419)
I20260924 20:20:31.969467 26658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:33523
I20260924 20:20:31.971369 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 26871
I20260924 20:20:31.971516 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:31.971840 19121 external_mini_cluster.cc:1428] Setting key 1f9049d744c03eb91a142f6a8829a96e
I20260924 20:20:31.976557 27008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.969053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.66:57377)
0924 20:20:31.969378 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:31.969391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:31.969484 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:31.969767 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:31.969774 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:31.970010 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:31.970132 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.970139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.970872 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.970875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.971739 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:31.971748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.972298 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.972301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:31.972402 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.973007 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:31.973021 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:31.973332 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:31.974983 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.974986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.974988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.975202 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.975496 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:31.975499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:31.975501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:31.975586 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:31.976102 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:31.976105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:31.976182 (+ 77us) client_negotiation.cc:769] Sending connection context
0924 20:20:31.976265 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:20:31.976370 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":109,"threads_started":1}
I20260924 20:20:31.976722 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:31.969155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57377 (local address 127.18.172.126:41343)
0924 20:20:31.969275 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:31.969279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:31.969462 (+ 183us) server_negotiation.cc:407] Connection header received
0924 20:20:31.969553 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:31.969556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:31.969593 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:31.969678 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:31.970248 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.970740 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.971847 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:31.972172 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:31.972277 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:31.973502 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:31.973506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:31.973507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:31.973527 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:31.974855 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.975303 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.975308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.975309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.975385 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:31.975676 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:31.975680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:31.975682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:31.975964 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:31.976064 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:31.976353 (+ 289us) server_negotiation.cc:299] Negotiation successful
0924 20:20:31.976489 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
WARNING: no policy specified for oryx/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
I20260924 20:20:32.002481 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41343
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:32.126575 27011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:32.126893 27011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:32.126979 27011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:32.131743 27011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:32.131832 27011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:32.131912 27011 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 20:20:32.132002 27011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:32.138070 27011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.18.172.126:41343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:32.139364 27011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:32.140525 27011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:32.150815 27019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:32.150810 27016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:32.150877 27017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:32.151034 27011 server_base.cc:1034] running on GCE node
I20260924 20:20:32.151648 27011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:32.152308 27011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:32.153479 27011 hybrid_clock.cc:648] HybridClock initialized: now 1790281232153462 us; error 29 us; skew 500 ppm
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281232, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:32.156243 27011 init.cc:377] Logged in from keytab as oryx/127.18.172.67@KRBTEST.COM (short username oryx)
I20260924 20:20:32.157080 27011 webserver.cc:460] Webserver started at http://127.18.172.67:44215/ using document root <none> and password file <none>
I20260924 20:20:32.157600 27011 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:32.157656 27011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:32.157814 27011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:32.159842 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
format_stamp: "Formatted at 2026-09-24 20:20:32 on dist-test-slave-m1qq"
server_key: "6fa273c4009eb2c2c421292fca2b0ba2"
server_key_iv: "790213ca224d77103ff311713934ad14"
server_key_version: "encryptionkey@0"
I20260924 20:20:32.160342 27011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
format_stamp: "Formatted at 2026-09-24 20:20:32 on dist-test-slave-m1qq"
server_key: "6fa273c4009eb2c2c421292fca2b0ba2"
server_key_iv: "790213ca224d77103ff311713934ad14"
server_key_version: "encryptionkey@0"
I20260924 20:20:32.164070 27011 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:20:32.166469 27026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:32.167924 27011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:32.168072 27011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
format_stamp: "Formatted at 2026-09-24 20:20:32 on dist-test-slave-m1qq"
server_key: "6fa273c4009eb2c2c421292fca2b0ba2"
server_key_iv: "790213ca224d77103ff311713934ad14"
server_key_version: "encryptionkey@0"
I20260924 20:20:32.168201 27011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:32.191496 27011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:32.194418 27011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:32.194619 27011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:32.196158 27011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:32.196257 27011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:32.196349 27011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:32.196400 27011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281232, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:32.207712 27011 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:37321
I20260924 20:20:32.207780 27141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:37321 every 8 connection(s)
I20260924 20:20:32.208851 27011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:32.209227 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27011
I20260924 20:20:32.209394 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:32.209740 19121 external_mini_cluster.cc:1428] Setting key 458859ee2ab498e8ee0b0305e0012188
I20260924 20:20:32.210821 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.197045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:47301 (local address 127.18.172.126:41343)
0924 20:20:32.197251 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.197258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.198101 (+ 843us) server_negotiation.cc:407] Connection header received
0924 20:20:32.199281 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.199285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.199344 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.199389 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:32.200934 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.201530 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.202929 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.203371 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.203520 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.206550 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:32.206554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:32.206558 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:32.206588 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:32.208509 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.209030 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.209042 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.209043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.209118 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.209561 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.209567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.209570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.209810 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:32.209921 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.210504 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.210631 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:20:32.211777 27035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.197490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.67:47301)
0924 20:20:32.197938 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.197986 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.199040 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.199786 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.199793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.200269 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:32.200749 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.200762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.201659 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.201661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.202793 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.202803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.203585 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.203592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.203801 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.204438 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:32.204474 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:32.206343 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:32.208624 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.208628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.208640 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.208882 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.209225 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.209228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.209233 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.209441 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.209916 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:32.209922 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:32.210224 (+ 302us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.210432 (+ 208us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.210766 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":140,"threads_started":1}
I20260924 20:20:32.213009 27142 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41343
I20260924 20:20:32.213156 27142 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:32.213732 27142 heartbeater.cc:511] Master 127.18.172.126:41343 requested a full tablet report, sending...
I20260924 20:20:32.214747 26658 ts_manager.cc:194] Registered new tserver with Master: f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:32.215528 26658 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:47301
I20260924 20:20:32.221858 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.215164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:41561 (local address 127.18.172.126:41343)
0924 20:20:32.215333 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.215336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.215404 (+ 68us) server_negotiation.cc:407] Connection header received
0924 20:20:32.215531 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.215533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.215566 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.215668 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:32.216129 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.216640 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.217463 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.217699 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.217822 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.218910 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:32.218913 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:32.218916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:32.218939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:32.220314 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.220730 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.220735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.220737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.220818 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.221086 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.221091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.221092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.221258 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:32.221362 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.221614 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.221726 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 20:20:32.221869 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.215075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.18.172.67:41561)
0924 20:20:32.215339 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.215353 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.215443 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.215696 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.215698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.215908 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:32.216025 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.216036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.216765 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.216768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.217344 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.217351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.217810 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.217814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.217905 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.218469 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:32.218480 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:32.218772 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:32.220433 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.220436 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.220438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.220636 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.220909 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.220913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.220915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.220999 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.221352 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:32.221355 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:32.221454 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.221560 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.221693 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":86,"threads_started":1}
I20260924 20:20:32.224092 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:20:32.248409 26708 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:32.241871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.18.172.126:41343)
0924 20:20:32.242036 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.242040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.242052 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:32.242093 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.242095 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.242133 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.242230 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:32.243094 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.243614 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.244558 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.244776 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.244893 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.248283 (+ 3390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56006: BlockingRecv error: recv got EOF from 127.0.0.1:56006 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:32.258987 26708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.250023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56022 (local address 127.18.172.126:41343)
0924 20:20:32.250146 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.250152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.250326 (+ 174us) server_negotiation.cc:407] Connection header received
0924 20:20:32.250424 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.250426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.250482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.250553 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:32.251260 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.251804 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.252813 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.253047 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.253135 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.255714 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:32.255717 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:32.255719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:32.255743 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:32.257241 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.257689 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.257692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.257693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.257768 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.258066 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.258071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.258074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.258302 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:32.258501 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.258701 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.258793 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:20:32.262413 26658 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:32.265897 26658 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:32.281481 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.275239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46533 (local address 127.0.0.1:35536)
0924 20:20:32.275616 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.275665 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.275793 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.277080 (+ 1287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.277086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.277109 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.277221 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.277228 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.278331 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.278337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.279675 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.279685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.280814 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.280818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.280957 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.281039 (+ 82us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.281145 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.281291 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":58,"thread_start_us":98,"threads_started":1}
I20260924 20:20:32.281494 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.275979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35536 (local address 127.18.172.65:46533)
0924 20:20:32.276563 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.276568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.276592 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:20:32.276673 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.276679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.276904 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.277386 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.277518 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.278215 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.279803 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.280677 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.281004 (+ 327us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.281155 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.281208 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.281354 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":87,"threads_started":1}
I20260924 20:20:32.281802 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.276378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.18.172.67:37321)
0924 20:20:32.276762 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.276772 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.276791 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:32.276854 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.276858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.277010 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.277147 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.277498 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.278612 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.279960 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.280649 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.280768 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.280944 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.281330 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.281549 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":101,"threads_started":1}
I20260924 20:20:32.282214 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.276031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51218 (local address 127.18.172.66:45419)
0924 20:20:32.276393 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.276405 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.276621 (+ 216us) server_negotiation.cc:407] Connection header received
0924 20:20:32.276702 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.276707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.276890 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.277052 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.277351 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.278597 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.280299 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.281083 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.281184 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.281319 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.281968 (+ 649us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.282066 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":116,"threads_started":1}
I20260924 20:20:32.282481 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.275587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:45419 (local address 127.0.0.1:51218)
0924 20:20:32.276431 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.276443 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.276527 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.277052 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.277058 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.277080 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.277212 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.277219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.279073 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.279076 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.280162 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.280172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.281376 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.281380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.281474 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.281497 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.281569 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.281646 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":21,"thread_start_us":77,"threads_started":1}
I20260924 20:20:32.283179 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.275316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.0.0.1:49898)
0924 20:20:32.276139 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.276154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.276246 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.277294 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.277297 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.277317 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.277387 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.277392 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.278724 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.278727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.279851 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.279858 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.280779 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.280782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.280914 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.281037 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.281146 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.281474 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":260,"thread_start_us":120,"threads_started":1}
I20260924 20:20:32.285049 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2fa838820e4ce0817675c082e4c035 (DEFAULT_TABLE table=test-table [id=877a618347f5416b845589bb7f5b829c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.285046 26796 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2fa838820e4ce0817675c082e4c035 (DEFAULT_TABLE table=test-table [id=877a618347f5416b845589bb7f5b829c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.286203 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2fa838820e4ce0817675c082e4c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.286203 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2fa838820e4ce0817675c082e4c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.289064 27076 tablet_service.cc:1467] Processing CreateTablet for tablet 1b2fa838820e4ce0817675c082e4c035 (DEFAULT_TABLE table=test-table [id=877a618347f5416b845589bb7f5b829c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.289983 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b2fa838820e4ce0817675c082e4c035. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.293749 27170 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Bootstrap starting.
I20260924 20:20:32.294989 27171 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Bootstrap starting.
I20260924 20:20:32.296175 27170 tablet_bootstrap.cc:654] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.296635 27172 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Bootstrap starting.
I20260924 20:20:32.296934 27171 tablet_bootstrap.cc:654] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.297742 27170 log.cc:824] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:32.297809 27171 log.cc:824] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:32.298754 27172 tablet_bootstrap.cc:654] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.299546 27172 log.cc:824] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:32.299669 27170 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: No bootstrap required, opened a new log
I20260924 20:20:32.299734 27171 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: No bootstrap required, opened a new log
I20260924 20:20:32.299888 27170 ts_tablet_manager.cc:1397] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:20:32.299914 27171 ts_tablet_manager.cc:1397] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:20:32.301426 27172 tablet_bootstrap.cc:492] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: No bootstrap required, opened a new log
I20260924 20:20:32.301628 27172 ts_tablet_manager.cc:1397] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260924 20:20:32.303612 27170 raft_consensus.cc:348] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.303895 27170 raft_consensus.cc:374] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.303967 27170 raft_consensus.cc:729] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.303893 27171 raft_consensus.cc:348] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.304158 27171 raft_consensus.cc:374] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.304236 27171 raft_consensus.cc:729] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.304474 27170 consensus_queue.cc:260] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.304664 27171 consensus_queue.cc:260] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.305292 26862 heartbeater.cc:503] Master 127.18.172.126:41343 was elected leader, sending a full tablet report...
I20260924 20:20:32.305549 27002 heartbeater.cc:503] Master 127.18.172.126:41343 was elected leader, sending a full tablet report...
I20260924 20:20:32.305789 27172 raft_consensus.cc:348] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.306028 27172 raft_consensus.cc:374] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.306082 27172 raft_consensus.cc:729] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.306483 27172 consensus_queue.cc:260] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.306802 27170 ts_tablet_manager.cc:1428] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260924 20:20:32.307322 27142 heartbeater.cc:503] Master 127.18.172.126:41343 was elected leader, sending a full tablet report...
I20260924 20:20:32.307538 27172 ts_tablet_manager.cc:1428] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260924 20:20:32.307824 27171 ts_tablet_manager.cc:1428] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260924 20:20:32.460639 27143 tablet.cc:2369] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:20:32.464047 27003 tablet.cc:2369] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:32.484469 27177 raft_consensus.cc:482] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:32.484683 27177 raft_consensus.cc:504] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
W20260924 20:20:32.485517 26863 tablet.cc:2369] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:20:32.485893 27177 leader_election.cc:288] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:32.489888 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.486518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:45755 (local address 127.18.172.67:37321)
0924 20:20:32.486642 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.486648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.486699 (+ 51us) server_negotiation.cc:407] Connection header received
0924 20:20:32.486758 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.486760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.486796 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.486873 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.487219 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.487747 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.488936 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.489312 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.489444 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.489498 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.489678 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.489756 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:20:32.489909 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.486200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:51307 (local address 127.18.172.65:46533)
0924 20:20:32.486341 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.486345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.486572 (+ 227us) server_negotiation.cc:407] Connection header received
0924 20:20:32.486752 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.486754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.486794 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.486880 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.487271 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.487761 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.488864 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.489315 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.489448 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.489500 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.489711 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.489786 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:20:32.489971 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.486187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.18.172.66:45755)
0924 20:20:32.486470 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.486481 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.486623 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.486898 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.486901 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.486923 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.487104 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.487118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.487859 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.487864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.488779 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.488820 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.489431 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.489435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.489538 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.489569 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.489640 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.489756 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":61,"spinlock_wait_cycles":7808,"thread_start_us":100,"threads_started":1}
I20260924 20:20:32.489971 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.486129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46533 (local address 127.18.172.66:51307)
0924 20:20:32.486457 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.486472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.486621 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.486902 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.486904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.486966 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.487137 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.487143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.487899 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.487901 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.488728 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.488737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.489452 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.489455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.489572 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.489588 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.489677 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.489794 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":67,"spinlock_wait_cycles":42752,"thread_start_us":68,"threads_started":1}
I20260924 20:20:32.490670 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2fa838820e4ce0817675c082e4c035" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" is_pre_election: true
I20260924 20:20:32.490671 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2fa838820e4ce0817675c082e4c035" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2" is_pre_election: true
I20260924 20:20:32.490969 27096 raft_consensus.cc:2454] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:32.490983 26816 raft_consensus.cc:2454] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:32.491511 26889 leader_election.cc:302] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, ce5666e3d55d480ab373d1e68e794024, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:32.491784 27177 raft_consensus.cc:2781] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:32.491868 27177 raft_consensus.cc:482] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:32.491909 27177 raft_consensus.cc:3037] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.492935 27177 raft_consensus.cc:504] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.493265 27177 leader_election.cc:288] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Requested vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:32.493621 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2fa838820e4ce0817675c082e4c035" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
I20260924 20:20:32.493643 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b2fa838820e4ce0817675c082e4c035" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2"
I20260924 20:20:32.493750 27096 raft_consensus.cc:3037] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.493798 26816 raft_consensus.cc:3037] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.494812 26816 raft_consensus.cc:2454] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:32.495143 26888 leader_election.cc:302] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, ce5666e3d55d480ab373d1e68e794024; no voters:
I20260924 20:20:32.495327 27177 raft_consensus.cc:2781] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:32.495332 27096 raft_consensus.cc:2454] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:32.495532 27177 raft_consensus.cc:686] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Becoming Leader. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Running, Role: LEADER
I20260924 20:20:32.495847 27177 consensus_queue.cc:237] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } }
I20260924 20:20:32.498946 26657 catalog_manager.cc:5495] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5666e3d55d480ab373d1e68e794024 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "ce5666e3d55d480ab373d1e68e794024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } health_report { overall_health: HEALTHY } } }
I20260924 20:20:32.516355 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.512553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51220 (local address 127.18.172.66:45419)
0924 20:20:32.512680 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.512684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.512923 (+ 239us) server_negotiation.cc:407] Connection header received
0924 20:20:32.513082 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.513084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.513139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.513230 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:32.513517 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.514044 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.514871 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.515149 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.515229 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.515533 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:32.515982 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:32.516067 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.516129 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.516177 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:20:32.522949 27096 raft_consensus.cc:1261] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:32.522992 26816 raft_consensus.cc:1261] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:32.523710 27177 consensus_queue.cc:1035] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:32.523933 27181 consensus_queue.cc:1035] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:32.530100 27184 mvcc.cc:205] Tried to move back new op lower bound from 7332991928406167552 to 7332991928305643520. Current Snapshot: MvccSnapshot[applied={T|T < 7332991928406167552}]
I20260924 20:20:32.531075 27185 mvcc.cc:205] Tried to move back new op lower bound from 7332991928406167552 to 7332991928305643520. Current Snapshot: MvccSnapshot[applied={T|T < 7332991928406167552}]
I20260924 20:20:32.538785 26657 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:32.538972 26657 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:32.541462 26657 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Sending DeleteTablet for 3 replicas of tablet 1b2fa838820e4ce0817675c082e4c035
I20260924 20:20:32.542194 27076 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2fa838820e4ce0817675c082e4c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:32 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:49898
I20260924 20:20:32.542253 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2fa838820e4ce0817675c082e4c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:32 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 20:20:32.542563 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b2fa838820e4ce0817675c082e4c035 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:32 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260924 20:20:32.543077 27195 tablet_replica.cc:330] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: stopping tablet replica
I20260924 20:20:32.543037 26657 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:32.543437 26657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:32.543438 27195 raft_consensus.cc:2229] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:32.543793 27195 raft_consensus.cc:2258] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:32.544374 27196 tablet_replica.cc:330] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: stopping tablet replica
I20260924 20:20:32.544644 27196 raft_consensus.cc:2229] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:32.544981 27196 raft_consensus.cc:2258] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:32.546376 27196 ts_tablet_manager.cc:1905] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:32.548518 27195 ts_tablet_manager.cc:1905] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:32.549070 26796 tablet_service.cc:1467] Processing CreateTablet for tablet 617d2eb9ee4943e5881899256673e022 (DEFAULT_TABLE table=test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.549321 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617d2eb9ee4943e5881899256673e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.549506 27196 ts_tablet_manager.cc:1918] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:32.549594 27196 log.cc:1196] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/1b2fa838820e4ce0817675c082e4c035
I20260924 20:20:32.549913 27196 ts_tablet_manager.cc:1939] T 1b2fa838820e4ce0817675c082e4c035 P ce5666e3d55d480ab373d1e68e794024: Deleting consensus metadata
I20260924 20:20:32.551225 27194 tablet_replica.cc:330] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: stopping tablet replica
I20260924 20:20:32.551342 27195 ts_tablet_manager.cc:1918] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:32.551424 27195 log.cc:1196] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/1b2fa838820e4ce0817675c082e4c035
I20260924 20:20:32.551533 27194 raft_consensus.cc:2229] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:32.551566 27170 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Bootstrap starting.
I20260924 20:20:32.551744 27195 ts_tablet_manager.cc:1939] T 1b2fa838820e4ce0817675c082e4c035 P 926db20805024cca9997a8d53c964fc2: Deleting consensus metadata
I20260924 20:20:32.551848 27194 raft_consensus.cc:2258] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:32.552508 27170 tablet_bootstrap.cc:654] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.553215 27194 ts_tablet_manager.cc:1905] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:32.553728 26644 catalog_manager.cc:4841] TS 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533): tablet 1b2fa838820e4ce0817675c082e4c035 (table test-table [id=877a618347f5416b845589bb7f5b829c]) successfully deleted
I20260924 20:20:32.553996 26646 catalog_manager.cc:4841] TS ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419): tablet 1b2fa838820e4ce0817675c082e4c035 (table test-table [id=877a618347f5416b845589bb7f5b829c]) successfully deleted
I20260924 20:20:32.554641 27076 tablet_service.cc:1467] Processing CreateTablet for tablet 617d2eb9ee4943e5881899256673e022 (DEFAULT_TABLE table=test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.554796 26936 tablet_service.cc:1467] Processing CreateTablet for tablet 617d2eb9ee4943e5881899256673e022 (DEFAULT_TABLE table=test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:32.554885 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617d2eb9ee4943e5881899256673e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.555702 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 617d2eb9ee4943e5881899256673e022. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.555745 27194 ts_tablet_manager.cc:1918] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:32.555827 27194 log.cc:1196] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/1b2fa838820e4ce0817675c082e4c035
I20260924 20:20:32.556139 27194 ts_tablet_manager.cc:1939] T 1b2fa838820e4ce0817675c082e4c035 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting consensus metadata
I20260924 20:20:32.557174 27172 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Bootstrap starting.
I20260924 20:20:32.558147 27172 tablet_bootstrap.cc:654] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.558823 27170 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: No bootstrap required, opened a new log
I20260924 20:20:32.558924 27170 ts_tablet_manager.cc:1397] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260924 20:20:32.558988 26645 catalog_manager.cc:4841] TS f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321): tablet 1b2fa838820e4ce0817675c082e4c035 (table test-table [id=877a618347f5416b845589bb7f5b829c]) successfully deleted
I20260924 20:20:32.559440 27172 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: No bootstrap required, opened a new log
I20260924 20:20:32.559427 27170 raft_consensus.cc:348] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.559508 27172 ts_tablet_manager.cc:1397] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:32.559518 27170 raft_consensus.cc:374] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.559540 27170 raft_consensus.cc:729] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.559674 27170 consensus_queue.cc:260] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.559937 27170 ts_tablet_manager.cc:1428] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.559890 27172 raft_consensus.cc:348] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.559995 27172 raft_consensus.cc:374] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.560025 27172 raft_consensus.cc:729] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.560155 27171 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Bootstrap starting.
I20260924 20:20:32.560158 27172 consensus_queue.cc:260] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.560384 27172 ts_tablet_manager.cc:1428] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.561141 27171 tablet_bootstrap.cc:654] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.562496 27171 tablet_bootstrap.cc:492] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: No bootstrap required, opened a new log
I20260924 20:20:32.562587 27171 ts_tablet_manager.cc:1397] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:32.563107 27171 raft_consensus.cc:348] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.563226 27171 raft_consensus.cc:374] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.563275 27171 raft_consensus.cc:729] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.563405 27171 consensus_queue.cc:260] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.563690 27171 ts_tablet_manager.cc:1428] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.733558 27176 raft_consensus.cc:482] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:32.733718 27176 raft_consensus.cc:504] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.734843 27176 leader_election.cc:288] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:32.740372 27200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.735258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.18.172.65:42363)
0924 20:20:32.735608 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.735624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.735749 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.736072 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.736075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.736090 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.736240 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.736248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.737384 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.737388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.739112 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.739123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.739896 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.739900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.740019 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.740043 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.740145 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.740228 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":203,"thread_start_us":84,"threads_started":1}
I20260924 20:20:32.740368 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.735387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:42363 (local address 127.18.172.67:37321)
0924 20:20:32.735560 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.735565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.735713 (+ 148us) server_negotiation.cc:407] Connection header received
0924 20:20:32.735898 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.735903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.735963 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.736062 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.736398 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.737205 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.739338 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.739752 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.739895 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.739952 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.740165 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.740238 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 20:20:32.740648 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.735245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33815 (local address 127.18.172.66:45419)
0924 20:20:32.735561 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.735566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.735800 (+ 234us) server_negotiation.cc:407] Connection header received
0924 20:20:32.736000 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.736003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.736057 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.736129 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.736490 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.737210 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.739258 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.739931 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.740066 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.740119 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.740446 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.740527 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260924 20:20:32.740994 27201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.735261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:45419 (local address 127.18.172.65:33815)
0924 20:20:32.735710 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.735724 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.735832 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.736182 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.736187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.736200 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.736353 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.736362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.737398 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.737400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.739112 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.739123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.740199 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.740203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.740312 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.740335 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.740791 (+ 456us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.740866 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":236,"thread_start_us":113,"threads_started":1}
I20260924 20:20:32.741003 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617d2eb9ee4943e5881899256673e022" candidate_uuid: "926db20805024cca9997a8d53c964fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" is_pre_election: true
I20260924 20:20:32.741153 27096 raft_consensus.cc:2454] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926db20805024cca9997a8d53c964fc2 in term 0.
I20260924 20:20:32.741488 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617d2eb9ee4943e5881899256673e022" candidate_uuid: "926db20805024cca9997a8d53c964fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5666e3d55d480ab373d1e68e794024" is_pre_election: true
I20260924 20:20:32.741564 26749 leader_election.cc:302] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:32.741729 26956 raft_consensus.cc:2454] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926db20805024cca9997a8d53c964fc2 in term 0.
I20260924 20:20:32.741832 27176 raft_consensus.cc:2781] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:32.741902 27176 raft_consensus.cc:482] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:32.741950 27176 raft_consensus.cc:3037] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.742988 27176 raft_consensus.cc:504] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.743441 27176 leader_election.cc:288] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [CANDIDATE]: Term 1 election: Requested vote from peers ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:32.743768 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617d2eb9ee4943e5881899256673e022" candidate_uuid: "926db20805024cca9997a8d53c964fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5666e3d55d480ab373d1e68e794024"
I20260924 20:20:32.743842 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "617d2eb9ee4943e5881899256673e022" candidate_uuid: "926db20805024cca9997a8d53c964fc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
I20260924 20:20:32.743911 26956 raft_consensus.cc:3037] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.743973 27096 raft_consensus.cc:3037] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.745088 27096 raft_consensus.cc:2454] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926db20805024cca9997a8d53c964fc2 in term 1.
I20260924 20:20:32.745146 26956 raft_consensus.cc:2454] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926db20805024cca9997a8d53c964fc2 in term 1.
I20260924 20:20:32.745455 26749 leader_election.cc:302] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:32.745630 27176 raft_consensus.cc:2781] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:32.745857 27176 raft_consensus.cc:686] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 1 LEADER]: Becoming Leader. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Running, Role: LEADER
I20260924 20:20:32.746115 27176 consensus_queue.cc:237] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.748445 26659 catalog_manager.cc:5495] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 926db20805024cca9997a8d53c964fc2 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "926db20805024cca9997a8d53c964fc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } health_report { overall_health: UNKNOWN } } }
Sep 24 20:20:32 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281231, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.126@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:32.821761 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.812937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.18.172.126:41343)
0924 20:20:32.813292 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.813302 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.813440 (+ 138us) server_negotiation.cc:407] Connection header received
0924 20:20:32.813584 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.813587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.813631 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.813721 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:32.814458 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.814987 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.815816 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.816040 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.816145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.818370 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:32.818373 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:32.818375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:32.818394 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:32.820052 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.820503 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.820508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.820510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.820588 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:32.820847 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:32.820852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:32.820854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:32.821025 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:32.821130 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.821400 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.821577 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260924 20:20:32.821761 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.812924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41343 (local address 127.0.0.1:56028)
0924 20:20:32.813350 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.813368 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.813499 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.813738 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.813741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.814162 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:32.814338 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.814346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.815140 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.815144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.815698 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.815705 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.816149 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.816152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.816228 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.816807 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:32.816819 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:32.818230 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:32.820160 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.820171 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.820178 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.820400 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.820678 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:32.820682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:32.820684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:32.820759 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:32.821130 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:32.821138 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:32.821242 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.821358 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.821516 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":113,"threads_started":1}
I20260924 20:20:32.825806 26659 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:56028:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 20:20:32.826529 26659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:32.831339 26936 tablet_service.cc:1467] Processing CreateTablet for tablet e4637d9b8e5b4f1f9c8750fa02002c93 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2300eaecb11d4281bca2b4118cdb1c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:32.831372 26796 tablet_service.cc:1467] Processing CreateTablet for tablet e4637d9b8e5b4f1f9c8750fa02002c93 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2300eaecb11d4281bca2b4118cdb1c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:32.831758 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4637d9b8e5b4f1f9c8750fa02002c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.831637 27076 tablet_service.cc:1467] Processing CreateTablet for tablet e4637d9b8e5b4f1f9c8750fa02002c93 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2300eaecb11d4281bca2b4118cdb1c93]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 20:20:32.831787 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4637d9b8e5b4f1f9c8750fa02002c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.831976 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4637d9b8e5b4f1f9c8750fa02002c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:32.834195 27170 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: Bootstrap starting.
I20260924 20:20:32.834769 27172 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: Bootstrap starting.
I20260924 20:20:32.835171 27170 tablet_bootstrap.cc:654] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.835508 27171 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: Bootstrap starting.
I20260924 20:20:32.835798 27172 tablet_bootstrap.cc:654] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.836207 27171 tablet_bootstrap.cc:654] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:32.836983 27170 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: No bootstrap required, opened a new log
I20260924 20:20:32.837021 27172 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: No bootstrap required, opened a new log
I20260924 20:20:32.837101 27172 ts_tablet_manager.cc:1397] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:32.837101 27170 ts_tablet_manager.cc:1397] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:32.837338 27171 tablet_bootstrap.cc:492] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: No bootstrap required, opened a new log
I20260924 20:20:32.837423 27171 ts_tablet_manager.cc:1397] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:32.837575 27170 raft_consensus.cc:348] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.837577 27172 raft_consensus.cc:348] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.837694 27172 raft_consensus.cc:374] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.837694 27170 raft_consensus.cc:374] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.837738 27172 raft_consensus.cc:729] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.837738 27170 raft_consensus.cc:729] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.837828 27171 raft_consensus.cc:348] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.837870 27172 consensus_queue.cc:260] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.837867 27170 consensus_queue.cc:260] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.837936 27171 raft_consensus.cc:374] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:32.837980 27171 raft_consensus.cc:729] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Initialized, Role: FOLLOWER
I20260924 20:20:32.838164 27172 ts_tablet_manager.cc:1428] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.838109 27171 consensus_queue.cc:260] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.838197 27170 ts_tablet_manager.cc:1428] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.838428 27171 ts_tablet_manager.cc:1428] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:32.838382 27178 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:32.838532 27178 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:32.838488 27176 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:32.838631 27176 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:32.838624 27191 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:32.838739 27191 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:32.957901 27178 raft_consensus.cc:482] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:32.958032 27178 raft_consensus.cc:504] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.959177 27178 leader_election.cc:288] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419)
I20260924 20:20:32.963402 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.959498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:60665 (local address 127.18.172.66:45419)
0924 20:20:32.959622 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.959627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.959866 (+ 239us) server_negotiation.cc:407] Connection header received
0924 20:20:32.960026 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.960029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.960078 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.960157 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.960467 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.960982 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.962315 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.962768 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.962913 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.962980 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.963192 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.963280 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:32.963557 27214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.959341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46533 (local address 127.18.172.67:39323)
0924 20:20:32.959669 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.959681 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.959771 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.960013 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.960016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.960033 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.960167 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.960174 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.960981 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.960984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.962200 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.962211 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.963097 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.963102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.963212 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.963230 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.963310 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.963381 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":24,"thread_start_us":86,"threads_started":1}
I20260924 20:20:32.963554 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.959500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:39323 (local address 127.18.172.65:46533)
0924 20:20:32.959622 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.959627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.959761 (+ 134us) server_negotiation.cc:407] Connection header received
0924 20:20:32.959860 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.959863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.959912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.959992 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:32.960281 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.960862 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.962348 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.962981 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.963114 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.963162 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.963349 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.963433 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:32.964121 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4637d9b8e5b4f1f9c8750fa02002c93" candidate_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2" is_pre_election: true
I20260924 20:20:32.964288 26816 raft_consensus.cc:2454] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ac4b591bf84399b5ce57a60ae12a99 in term 0.
I20260924 20:20:32.964696 27215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.959341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:45419 (local address 127.18.172.67:60665)
0924 20:20:32.959781 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.959797 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.959898 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.960181 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.960183 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.960196 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:32.960328 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.960335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.961105 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.961107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.962164 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.962173 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.962905 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.962908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.963045 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.963075 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.964476 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.964555 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":44,"thread_start_us":65,"threads_started":1}
I20260924 20:20:32.964635 27028 leader_election.cc:302] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:32.964912 27178 raft_consensus.cc:2781] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:32.964984 27178 raft_consensus.cc:482] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:32.965027 27178 raft_consensus.cc:3037] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.965150 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4637d9b8e5b4f1f9c8750fa02002c93" candidate_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5666e3d55d480ab373d1e68e794024" is_pre_election: true
I20260924 20:20:32.965269 26956 raft_consensus.cc:2454] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7ac4b591bf84399b5ce57a60ae12a99 in term 0.
I20260924 20:20:32.966557 27178 raft_consensus.cc:504] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.966933 27178 leader_election.cc:288] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [CANDIDATE]: Term 1 election: Requested vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419)
I20260924 20:20:32.967301 26956 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4637d9b8e5b4f1f9c8750fa02002c93" candidate_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce5666e3d55d480ab373d1e68e794024"
I20260924 20:20:32.967329 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4637d9b8e5b4f1f9c8750fa02002c93" candidate_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2"
I20260924 20:20:32.967419 26956 raft_consensus.cc:3037] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.967470 26816 raft_consensus.cc:3037] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:32.968547 26956 raft_consensus.cc:2454] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ac4b591bf84399b5ce57a60ae12a99 in term 1.
I20260924 20:20:32.968643 26816 raft_consensus.cc:2454] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7ac4b591bf84399b5ce57a60ae12a99 in term 1.
I20260924 20:20:32.968875 27030 leader_election.cc:302] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce5666e3d55d480ab373d1e68e794024, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:32.969085 27178 raft_consensus.cc:2781] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:32.969257 27178 raft_consensus.cc:686] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 LEADER]: Becoming Leader. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Running, Role: LEADER
I20260924 20:20:32.969527 27178 consensus_queue.cc:237] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:32.970750 27216 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: TxnStatusTablet state changed. Reason: New leader f7ac4b591bf84399b5ce57a60ae12a99. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:32.970927 27216 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:32.971939 26658 catalog_manager.cc:5495] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } health_report { overall_health: HEALTHY } } }
I20260924 20:20:32.980295 27218 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 20:20:32.980448 27218 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 20:20:32.997157 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.992911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.0.0.1:49900)
0924 20:20:32.993044 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:32.993064 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:32.993152 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:32.993509 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:32.993512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:32.993519 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:32.993611 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.993615 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.994710 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.994713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.995391 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:32.995399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.996025 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.996027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:32.996090 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.996132 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:32.996909 (+ 777us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:32.996925 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 20:20:32.996978 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 20:20:32.997046 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 20:20:32.997197 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:32.993020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.18.172.67:37321)
0924 20:20:32.993168 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:32.993173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:32.993221 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:20:32.993302 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:32.993305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:32.993378 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:32.993493 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:32.993722 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.994599 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.995512 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:32.995905 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:32.996014 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:32.996266 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:32.996805 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:32.996935 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:32.996995 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 20:20:32.997055 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:20:33.000778 26816 raft_consensus.cc:1261] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Refusing update from remote peer f7ac4b591bf84399b5ce57a60ae12a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.001016 26956 raft_consensus.cc:1261] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Refusing update from remote peer f7ac4b591bf84399b5ce57a60ae12a99: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.001427 27216 consensus_queue.cc:1035] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.001617 27178 consensus_queue.cc:1035] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.004515 27176 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: TxnStatusTablet state changed. Reason: New leader f7ac4b591bf84399b5ce57a60ae12a99. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.004642 27176 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:33.004793 27216 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.004899 27216 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:33.005399 27186 mvcc.cc:205] Tried to move back new op lower bound from 7332991930366222336 to 7332991930245578752. Current Snapshot: MvccSnapshot[applied={T|T < 7332991930366222336 or (T in {7332991930366222336})}]
I20260924 20:20:33.006644 27191 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: TxnStatusTablet state changed. Reason: New leader f7ac4b591bf84399b5ce57a60ae12a99. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.006763 27191 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:33.007325 27176 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.007470 27176 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P 926db20805024cca9997a8d53c964fc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:33.009441 27178 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.009573 27178 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P f7ac4b591bf84399b5ce57a60ae12a99: This TxnStatusTablet replica's current role is: LEADER
I20260924 20:20:33.011343 27191 tablet_replica.cc:439] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } } }
I20260924 20:20:33.011466 27191 tablet_replica.cc:442] T e4637d9b8e5b4f1f9c8750fa02002c93 P ce5666e3d55d480ab373d1e68e794024: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 20:20:33.016391 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.012212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35544 (local address 127.18.172.65:46533)
0924 20:20:33.012322 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.012328 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.012343 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:20:33.012430 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.012435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.012505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.012615 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.012881 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.013822 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.014656 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.015074 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.015188 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.015419 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.016009 (+ 590us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.016134 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.016178 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.016243 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 20:20:33.027999 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.023599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:56163 (local address 127.18.172.67:37321)
0924 20:20:33.023726 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.023729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.023994 (+ 265us) server_negotiation.cc:407] Connection header received
0924 20:20:33.024129 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.024131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.024173 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.024235 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.025010 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.025514 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.026875 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.027155 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.027236 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.027580 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.027648 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.027702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.027838 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.027881 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:33.028090 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.023510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.18.172.65:56163)
0924 20:20:33.023905 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:33.023922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:33.024036 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:33.024276 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:33.024280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:33.024288 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:33.024833 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.024842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.025625 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.025629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.026752 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.026763 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.027267 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.027270 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.027377 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.027438 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.027736 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.027743 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:20:33.027823 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 20:20:33.027907 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":110,"threads_started":1}
I20260924 20:20:33.035387 27096 raft_consensus.cc:1261] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Refusing update from remote peer 926db20805024cca9997a8d53c964fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.035406 26956 raft_consensus.cc:1261] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Refusing update from remote peer 926db20805024cca9997a8d53c964fc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.035885 27202 consensus_queue.cc:1035] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.036144 27176 consensus_queue.cc:1035] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.058660 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.054325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46533 (local address 127.18.172.67:56191)
0924 20:20:33.054608 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:33.054620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:33.054737 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:33.054986 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:33.054989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:33.054996 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:33.055160 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.055166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.056334 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.056339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.057395 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.057408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.057847 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.057851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.057959 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.058018 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.058348 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.058357 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 20:20:33.058423 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 20:20:33.058504 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":96,"threads_started":1}
I20260924 20:20:33.058642 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.054360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:56191 (local address 127.18.172.65:46533)
0924 20:20:33.054485 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.054491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.054689 (+ 198us) server_negotiation.cc:407] Connection header received
0924 20:20:33.054839 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.054842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.054893 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.054976 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.055312 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.056179 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.057521 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.057734 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.057812 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.058164 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.058249 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.058344 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.058459 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.058504 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:20:33.085688 26659 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.085860 26659 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.088173 26659 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Sending DeleteTablet for 3 replicas of tablet 617d2eb9ee4943e5881899256673e022
I20260924 20:20:33.088883 27076 tablet_service.cc:1514] Processing DeleteTablet for tablet 617d2eb9ee4943e5881899256673e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:49898
I20260924 20:20:33.088913 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet 617d2eb9ee4943e5881899256673e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 20:20:33.089437 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet 617d2eb9ee4943e5881899256673e022 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260924 20:20:33.089789 27236 tablet_replica.cc:330] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: stopping tablet replica
I20260924 20:20:33.089957 27236 raft_consensus.cc:2229] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.090127 27236 raft_consensus.cc:2258] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.090911 27236 ts_tablet_manager.cc:1905] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.094339 27236 ts_tablet_manager.cc:1918] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:33.094496 27236 log.cc:1196] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/617d2eb9ee4943e5881899256673e022
I20260924 20:20:33.094853 27236 ts_tablet_manager.cc:1939] T 617d2eb9ee4943e5881899256673e022 P ce5666e3d55d480ab373d1e68e794024: Deleting consensus metadata
I20260924 20:20:33.095638 26646 catalog_manager.cc:4841] TS ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419): tablet 617d2eb9ee4943e5881899256673e022 (table test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]) successfully deleted
I20260924 20:20:33.097820 27235 tablet_replica.cc:330] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: stopping tablet replica
I20260924 20:20:33.097973 27235 raft_consensus.cc:2229] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:33.098310 27235 raft_consensus.cc:2258] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.099063 27235 ts_tablet_manager.cc:1905] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.099939 27234 tablet_replica.cc:330] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: stopping tablet replica
I20260924 20:20:33.100080 27234 raft_consensus.cc:2229] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.100210 27234 raft_consensus.cc:2258] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.101154 27234 ts_tablet_manager.cc:1905] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.102131 27235 ts_tablet_manager.cc:1918] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:33.102258 27235 log.cc:1196] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/617d2eb9ee4943e5881899256673e022
I20260924 20:20:33.102592 27235 ts_tablet_manager.cc:1939] T 617d2eb9ee4943e5881899256673e022 P 926db20805024cca9997a8d53c964fc2: Deleting consensus metadata
I20260924 20:20:33.103336 26644 catalog_manager.cc:4841] TS 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533): tablet 617d2eb9ee4943e5881899256673e022 (table test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]) successfully deleted
I20260924 20:20:33.104135 27234 ts_tablet_manager.cc:1918] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:33.104221 27234 log.cc:1196] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/617d2eb9ee4943e5881899256673e022
I20260924 20:20:33.104506 27234 ts_tablet_manager.cc:1939] T 617d2eb9ee4943e5881899256673e022 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting consensus metadata
I20260924 20:20:33.105242 26645 catalog_manager.cc:4841] TS f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321): tablet 617d2eb9ee4943e5881899256673e022 (table test-table [id=ef064adeb5ae4fb1b6aa3115a892490d]) successfully deleted
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[26614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:20:33.132220 27209 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:33.123673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.18.172.126:41343)
0924 20:20:33.123834 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.123840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.124045 (+ 205us) server_negotiation.cc:407] Connection header received
0924 20:20:33.124187 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.124191 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.124251 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.124355 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:33.125223 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.126189 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.127541 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.127959 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.128085 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.132086 (+ 4001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56040: BlockingRecv error: recv got EOF from 127.0.0.1:56040 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[26614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281233, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM
I20260924 20:20:33.145309 27209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.134581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.18.172.126:41343)
0924 20:20:33.134753 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.134760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.134912 (+ 152us) server_negotiation.cc:407] Connection header received
0924 20:20:33.135035 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.135038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.135095 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.135190 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:33.135900 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.136752 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.137895 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.138305 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.138419 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.141432 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:33.141437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:33.141440 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:33.141471 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:33.143402 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:33.143894 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:33.143898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:33.143900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:33.144010 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:33.144320 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:33.144324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:33.144326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:33.144572 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:33.144721 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.144968 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.145112 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 20:20:33.148233 26659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:33.148712 26659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:33.153934 26936 tablet_service.cc:1467] Processing CreateTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 (DEFAULT_TABLE table=test-table [id=e6a90f13118143c2b8d1f40eab5befb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.154248 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfa15b215b94151a2d17eeb8af6bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.155390 27076 tablet_service.cc:1467] Processing CreateTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 (DEFAULT_TABLE table=test-table [id=e6a90f13118143c2b8d1f40eab5befb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.155718 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfa15b215b94151a2d17eeb8af6bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.156793 27171 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Bootstrap starting.
I20260924 20:20:33.157676 27171 tablet_bootstrap.cc:654] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.159118 27171 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: No bootstrap required, opened a new log
I20260924 20:20:33.159201 27171 ts_tablet_manager.cc:1397] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:33.159376 26796 tablet_service.cc:1467] Processing CreateTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 (DEFAULT_TABLE table=test-table [id=e6a90f13118143c2b8d1f40eab5befb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.159677 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdfa15b215b94151a2d17eeb8af6bd59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.159621 27171 raft_consensus.cc:348] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.159730 27171 raft_consensus.cc:374] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.159760 27171 raft_consensus.cc:729] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.159873 27171 consensus_queue.cc:260] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.160111 27171 ts_tablet_manager.cc:1428] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:33.162474 27170 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Bootstrap starting.
I20260924 20:20:33.163448 27170 tablet_bootstrap.cc:654] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.165072 27172 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Bootstrap starting.
I20260924 20:20:33.166101 27172 tablet_bootstrap.cc:654] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.167631 27172 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: No bootstrap required, opened a new log
I20260924 20:20:33.167734 27172 ts_tablet_manager.cc:1397] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:33.168232 27172 raft_consensus.cc:348] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.168375 27172 raft_consensus.cc:374] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.168417 27172 raft_consensus.cc:729] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.168540 27172 consensus_queue.cc:260] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.168789 27172 ts_tablet_manager.cc:1428] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:33.171721 27170 tablet_bootstrap.cc:492] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: No bootstrap required, opened a new log
I20260924 20:20:33.171818 27170 ts_tablet_manager.cc:1397] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20260924 20:20:33.172264 27170 raft_consensus.cc:348] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.172372 27170 raft_consensus.cc:374] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.172401 27170 raft_consensus.cc:729] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.172536 27170 consensus_queue.cc:260] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.172780 27170 ts_tablet_manager.cc:1428] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:20:33.188596 27191 raft_consensus.cc:482] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:33.188751 27191 raft_consensus.cc:504] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.189159 27191 leader_election.cc:288] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:33.189556 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfa15b215b94151a2d17eeb8af6bd59" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2" is_pre_election: true
I20260924 20:20:33.189625 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfa15b215b94151a2d17eeb8af6bd59" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" is_pre_election: true
I20260924 20:20:33.189689 26816 raft_consensus.cc:2454] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:33.189755 27096 raft_consensus.cc:2454] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:33.189960 26888 leader_election.cc:302] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 926db20805024cca9997a8d53c964fc2, ce5666e3d55d480ab373d1e68e794024; no voters:
I20260924 20:20:33.190155 27191 raft_consensus.cc:2781] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:33.190208 27191 raft_consensus.cc:482] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:33.190248 27191 raft_consensus.cc:3037] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.191612 27191 raft_consensus.cc:504] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.191982 27191 leader_election.cc:288] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Requested vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:33.192337 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfa15b215b94151a2d17eeb8af6bd59" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2"
I20260924 20:20:33.192389 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdfa15b215b94151a2d17eeb8af6bd59" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
I20260924 20:20:33.192461 26816 raft_consensus.cc:3037] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.192493 27096 raft_consensus.cc:3037] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.193571 27096 raft_consensus.cc:2454] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:33.193609 26816 raft_consensus.cc:2454] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:33.193868 26889 leader_election.cc:302] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce5666e3d55d480ab373d1e68e794024, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:33.194065 27191 raft_consensus.cc:2781] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:33.194226 27191 raft_consensus.cc:686] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Becoming Leader. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Running, Role: LEADER
I20260924 20:20:33.194368 27191 consensus_queue.cc:237] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.196007 26659 catalog_manager.cc:5495] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5666e3d55d480ab373d1e68e794024 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "ce5666e3d55d480ab373d1e68e794024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:33.213744 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.210106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51222 (local address 127.18.172.66:45419)
0924 20:20:33.210216 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.210220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.210303 (+ 83us) server_negotiation.cc:407] Connection header received
0924 20:20:33.210458 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.210461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.210514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.210595 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.210847 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.211451 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.212649 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.212935 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.213050 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.213298 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.213375 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.213457 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.213554 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.213605 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:20:33.215651 26816 raft_consensus.cc:1261] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.215677 27096 raft_consensus.cc:1261] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.216055 27252 consensus_queue.cc:1035] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.216205 27191 consensus_queue.cc:1035] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.225610 26658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.225785 26658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.228005 26658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Sending DeleteTablet for 3 replicas of tablet cdfa15b215b94151a2d17eeb8af6bd59
I20260924 20:20:33.229272 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 20:20:33.229282 27076 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:49898
I20260924 20:20:33.229457 27234 tablet_replica.cc:330] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: stopping tablet replica
I20260924 20:20:33.229480 27235 tablet_replica.cc:330] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: stopping tablet replica
I20260924 20:20:33.229557 27234 raft_consensus.cc:2229] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.229640 27235 raft_consensus.cc:2229] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.229691 27234 raft_consensus.cc:2258] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.229744 27235 pending_rounds.cc:62] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Trying to abort 2 pending ops.
I20260924 20:20:33.229804 27235 pending_rounds.cc:68] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7332991931247357952 op_type: WRITE_OP write_request { tablet_id: "cdfa15b215b94151a2d17eeb8af6bd59" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\275\216\326\325\006\"8\n\ntest-admin\022*\n e6a90f13118143c2b8d1f40eab5befb4\020\001\030\001 \001(\001" signature: "<redacted>""y\370\320\330\t\257~\024T\021\377\302Z63\004`p\330Iw\006J\377\263&or\253=\372\365\211\224WD<\240\371Gm\r\210\r\206\304\237\004i\371\'\257s ~\240\246\305\031&\261\245:]" signing_key_seq_num: 0 } } request_id { client_id: "b9e236d2b37c4e53b3712351031215c3" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260924 20:20:33.230192 27235 pending_rounds.cc:68] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7332991931165720576 op_type: NO_OP noop_request { }
I20260924 20:20:33.230278 27235 raft_consensus.cc:2866] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260924 20:20:33.230273 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet cdfa15b215b94151a2d17eeb8af6bd59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260924 20:20:33.230342 27235 raft_consensus.cc:2258] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.230541 27236 tablet_replica.cc:330] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: stopping tablet replica
I20260924 20:20:33.230664 27236 raft_consensus.cc:2229] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:33.230851 27236 raft_consensus.cc:2258] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.231536 26659 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 20:20:33.231691 27236 ts_tablet_manager.cc:1905] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:20:33.231959 26659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:33.233168 27234 ts_tablet_manager.cc:1905] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.234305 27236 ts_tablet_manager.cc:1918] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:33.234386 27236 log.cc:1196] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/cdfa15b215b94151a2d17eeb8af6bd59
I20260924 20:20:33.234656 27236 ts_tablet_manager.cc:1939] T cdfa15b215b94151a2d17eeb8af6bd59 P ce5666e3d55d480ab373d1e68e794024: Deleting consensus metadata
I20260924 20:20:33.235476 27235 ts_tablet_manager.cc:1905] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.235535 27234 ts_tablet_manager.cc:1918] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:33.235636 27234 log.cc:1196] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/cdfa15b215b94151a2d17eeb8af6bd59
I20260924 20:20:33.235872 27234 ts_tablet_manager.cc:1939] T cdfa15b215b94151a2d17eeb8af6bd59 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting consensus metadata
I20260924 20:20:33.236204 26646 catalog_manager.cc:4841] TS ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419): tablet cdfa15b215b94151a2d17eeb8af6bd59 (table test-table [id=e6a90f13118143c2b8d1f40eab5befb4]) successfully deleted
I20260924 20:20:33.238088 27235 ts_tablet_manager.cc:1918] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:33.238175 27235 log.cc:1196] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/cdfa15b215b94151a2d17eeb8af6bd59
I20260924 20:20:33.238440 27235 ts_tablet_manager.cc:1939] T cdfa15b215b94151a2d17eeb8af6bd59 P 926db20805024cca9997a8d53c964fc2: Deleting consensus metadata
I20260924 20:20:33.239181 26645 catalog_manager.cc:4841] TS f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321): tablet cdfa15b215b94151a2d17eeb8af6bd59 (table test-table [id=e6a90f13118143c2b8d1f40eab5befb4]) successfully deleted
I20260924 20:20:33.239964 26644 catalog_manager.cc:4841] TS 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533): tablet cdfa15b215b94151a2d17eeb8af6bd59 (table test-table [id=e6a90f13118143c2b8d1f40eab5befb4]) successfully deleted
I20260924 20:20:33.240499 26796 tablet_service.cc:1467] Processing CreateTablet for tablet b714e7302e68461cb97988dea02e7fc1 (DEFAULT_TABLE table=test-table [id=81f52b2fea4e4c449bc90084dabc7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.240677 27076 tablet_service.cc:1467] Processing CreateTablet for tablet b714e7302e68461cb97988dea02e7fc1 (DEFAULT_TABLE table=test-table [id=81f52b2fea4e4c449bc90084dabc7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.240747 26796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b714e7302e68461cb97988dea02e7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.240896 27076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b714e7302e68461cb97988dea02e7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.242959 27170 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: Bootstrap starting.
I20260924 20:20:33.243208 27172 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Bootstrap starting.
I20260924 20:20:33.243899 27172 tablet_bootstrap.cc:654] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.243896 27170 tablet_bootstrap.cc:654] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.245085 27172 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: No bootstrap required, opened a new log
I20260924 20:20:33.245088 26936 tablet_service.cc:1467] Processing CreateTablet for tablet b714e7302e68461cb97988dea02e7fc1 (DEFAULT_TABLE table=test-table [id=81f52b2fea4e4c449bc90084dabc7d1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:33.245167 27172 ts_tablet_manager.cc:1397] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:33.245337 26936 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b714e7302e68461cb97988dea02e7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:33.245584 27172 raft_consensus.cc:348] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.245698 27172 raft_consensus.cc:374] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.245731 27172 raft_consensus.cc:729] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7ac4b591bf84399b5ce57a60ae12a99, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.245846 27172 consensus_queue.cc:260] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.246071 27172 ts_tablet_manager.cc:1428] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:33.247799 27171 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Bootstrap starting.
I20260924 20:20:33.248660 27171 tablet_bootstrap.cc:654] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:33.249764 27171 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: No bootstrap required, opened a new log
I20260924 20:20:33.249876 27171 ts_tablet_manager.cc:1397] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:33.250293 27171 raft_consensus.cc:348] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.250412 27171 raft_consensus.cc:374] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.250460 27171 raft_consensus.cc:729] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.250603 27171 consensus_queue.cc:260] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.250876 27171 ts_tablet_manager.cc:1428] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:20:33.251431 27170 tablet_bootstrap.cc:492] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: No bootstrap required, opened a new log
I20260924 20:20:33.251521 27170 ts_tablet_manager.cc:1397] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260924 20:20:33.252018 27170 raft_consensus.cc:348] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.252107 27170 raft_consensus.cc:374] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:33.252128 27170 raft_consensus.cc:729] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926db20805024cca9997a8d53c964fc2, State: Initialized, Role: FOLLOWER
I20260924 20:20:33.252260 27170 consensus_queue.cc:260] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.252538 27170 ts_tablet_manager.cc:1428] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260924 20:20:33.325001 27191 raft_consensus.cc:482] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:33.325145 27191 raft_consensus.cc:504] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.325780 27191 leader_election.cc:288] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:33.326289 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b714e7302e68461cb97988dea02e7fc1" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2" is_pre_election: true
I20260924 20:20:33.326289 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b714e7302e68461cb97988dea02e7fc1" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" is_pre_election: true
I20260924 20:20:33.326463 26816 raft_consensus.cc:2454] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:33.326483 27096 raft_consensus.cc:2454] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 0.
I20260924 20:20:33.326768 26889 leader_election.cc:302] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce5666e3d55d480ab373d1e68e794024, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:33.326948 27191 raft_consensus.cc:2781] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:33.327047 27191 raft_consensus.cc:482] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:33.327101 27191 raft_consensus.cc:3037] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.328405 27191 raft_consensus.cc:504] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.328838 27191 leader_election.cc:288] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Requested vote from peers 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533), f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321)
I20260924 20:20:33.329155 26816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b714e7302e68461cb97988dea02e7fc1" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "926db20805024cca9997a8d53c964fc2"
I20260924 20:20:33.329279 26816 raft_consensus.cc:3037] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.329252 27096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b714e7302e68461cb97988dea02e7fc1" candidate_uuid: "ce5666e3d55d480ab373d1e68e794024" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7ac4b591bf84399b5ce57a60ae12a99"
I20260924 20:20:33.329375 27096 raft_consensus.cc:3037] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:33.330440 26816 raft_consensus.cc:2454] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:33.330452 27096 raft_consensus.cc:2454] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5666e3d55d480ab373d1e68e794024 in term 1.
I20260924 20:20:33.330751 26889 leader_election.cc:302] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce5666e3d55d480ab373d1e68e794024, f7ac4b591bf84399b5ce57a60ae12a99; no voters:
I20260924 20:20:33.330937 27191 raft_consensus.cc:2781] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:33.331080 27191 raft_consensus.cc:686] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Becoming Leader. State: Replica: ce5666e3d55d480ab373d1e68e794024, State: Running, Role: LEADER
I20260924 20:20:33.331233 27191 consensus_queue.cc:237] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } }
I20260924 20:20:33.332772 26658 catalog_manager.cc:5495] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5666e3d55d480ab373d1e68e794024 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "ce5666e3d55d480ab373d1e68e794024" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5666e3d55d480ab373d1e68e794024" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 45419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:33.367231 27169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.363664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:60037 (local address 127.18.172.67:37321)
0924 20:20:33.363833 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.363838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.364005 (+ 167us) server_negotiation.cc:407] Connection header received
0924 20:20:33.364176 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.364182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.364231 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.364317 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.364612 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.365133 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.366028 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.366313 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.366420 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.366747 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.366829 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.366905 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.367033 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.367085 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:20:33.367234 27264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.363505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37321 (local address 127.18.172.66:60037)
0924 20:20:33.363903 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:33.363918 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:33.364071 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:33.364329 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:33.364332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:33.364340 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:33.364465 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.364491 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.365246 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.365250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.365904 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.365913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.366442 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.366445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.366540 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.366585 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.366930 (+ 345us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.366937 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:20:33.367022 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:20:33.367108 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":139,"threads_started":1}
I20260924 20:20:33.373133 27096 raft_consensus.cc:1261] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.373250 26816 raft_consensus.cc:1261] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Refusing update from remote peer ce5666e3d55d480ab373d1e68e794024: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:33.373637 27191 consensus_queue.cc:1035] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7ac4b591bf84399b5ce57a60ae12a99" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.373812 27255 consensus_queue.cc:1035] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [LEADER]: Connected to new peer: Peer: permanent_uuid: "926db20805024cca9997a8d53c964fc2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:33.390650 27230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.387480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:45419 (local address 127.18.172.67:52543)
0924 20:20:33.387602 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:33.387618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:33.387702 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:33.387987 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:33.387989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:33.387996 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 20:20:33.388084 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.388107 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.389017 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.389020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.389634 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:33.389643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.390029 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.390031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:33.390090 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.390116 (+ 26us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.390388 (+ 272us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.390396 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:20:33.390456 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 20:20:33.390540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 20:20:33.390679 27168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.387543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:52543 (local address 127.18.172.66:45419)
0924 20:20:33.387735 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:33.387741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:33.387774 (+ 33us) server_negotiation.cc:407] Connection header received
0924 20:20:33.387814 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:33.387817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:33.387864 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:33.387960 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:33.388201 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.388918 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.389736 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:33.389941 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:33.390047 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:33.390247 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:33.390312 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:33.390374 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:33.390504 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 20:20:33.390555 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260924 20:20:33.411806 26658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.411996 26658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:33.414026 26658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ac9ad9563b54d05b39a817e569363f7: Sending DeleteTablet for 3 replicas of tablet b714e7302e68461cb97988dea02e7fc1
I20260924 20:20:33.414575 26796 tablet_service.cc:1514] Processing DeleteTablet for tablet b714e7302e68461cb97988dea02e7fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35536
I20260924 20:20:33.414645 26936 tablet_service.cc:1514] Processing DeleteTablet for tablet b714e7302e68461cb97988dea02e7fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51218
I20260924 20:20:33.414767 27235 tablet_replica.cc:330] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2: stopping tablet replica
I20260924 20:20:33.414821 27236 tablet_replica.cc:330] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: stopping tablet replica
I20260924 20:20:33.414871 27235 raft_consensus.cc:2229] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.414942 27236 raft_consensus.cc:2229] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:20:33.414983 27235 raft_consensus.cc:2258] T b714e7302e68461cb97988dea02e7fc1 P 926db20805024cca9997a8d53c964fc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.415164 27236 raft_consensus.cc:2258] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.415225 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 26731
I20260924 20:20:33.416531 27076 tablet_service.cc:1514] Processing DeleteTablet for tablet b714e7302e68461cb97988dea02e7fc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:33 UTC) from {username='oryx', principal='oryx/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:49898
I20260924 20:20:33.416724 27234 tablet_replica.cc:330] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: stopping tablet replica
I20260924 20:20:33.416857 27234 raft_consensus.cc:2229] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:33.416996 27234 raft_consensus.cc:2258] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:33.417557 27236 ts_tablet_manager.cc:1905] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.417793 27234 ts_tablet_manager.cc:1905] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:33.419587 27236 ts_tablet_manager.cc:1918] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:33.419641 27236 log.cc:1196] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/b714e7302e68461cb97988dea02e7fc1
I20260924 20:20:33.419821 27236 ts_tablet_manager.cc:1939] T b714e7302e68461cb97988dea02e7fc1 P ce5666e3d55d480ab373d1e68e794024: Deleting consensus metadata
I20260924 20:20:33.420517 26646 catalog_manager.cc:4841] TS ce5666e3d55d480ab373d1e68e794024 (127.18.172.66:45419): tablet b714e7302e68461cb97988dea02e7fc1 (table test-table [id=81f52b2fea4e4c449bc90084dabc7d1f]) successfully deleted
I20260924 20:20:33.420873 27234 ts_tablet_manager.cc:1918] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 20:20:33.420969 27234 log.cc:1196] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/b714e7302e68461cb97988dea02e7fc1
I20260924 20:20:33.421247 27234 ts_tablet_manager.cc:1939] T b714e7302e68461cb97988dea02e7fc1 P f7ac4b591bf84399b5ce57a60ae12a99: Deleting consensus metadata
I20260924 20:20:33.421928 26645 catalog_manager.cc:4841] TS f7ac4b591bf84399b5ce57a60ae12a99 (127.18.172.67:37321): tablet b714e7302e68461cb97988dea02e7fc1 (table test-table [id=81f52b2fea4e4c449bc90084dabc7d1f]) successfully deleted
W20260924 20:20:33.423789 26644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:46533 (error 108)
I20260924 20:20:33.423965 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 26871
I20260924 20:20:33.431128 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27011
I20260924 20:20:33.432818 27269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:33.431485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46533 (local address 127.0.0.1:35554)
0924 20:20:33.432586 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:33.432678 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":84,"threads_started":1}
W20260924 20:20:33.433004 26644 catalog_manager.cc:4584] TS 926db20805024cca9997a8d53c964fc2 (127.18.172.65:46533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b714e7302e68461cb97988dea02e7fc1: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46533: connect: Connection refused (error 111)
I20260924 20:20:33.437909 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 26627
2026-09-24T20:20:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3410 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[27272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[27272](info): set up 2 sockets
Sep 24 20:20:33 dist-test-slave-m1qq krb5kdc[27272](info): commencing operation
krb5kdc: starting...
W20260924 20:20:34.501322 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:34 dist-test-slave-m1qq krb5kdc[27272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:34.626139 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:46761
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:46761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:34.740586 27285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:34.740999 27285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:34.741098 27285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:34.745642 27285 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:34.745747 27285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:34.745785 27285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:34.745826 27285 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:34.745867 27285 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:34.751617 27285 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:46761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:34.752799 27285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:34.753861 27285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:34.762818 27293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:34.762823 27291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:34.762806 27290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:34.763084 27285 server_base.cc:1034] running on GCE node
I20260924 20:20:34.763775 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:34.764998 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:34.766196 27285 hybrid_clock.cc:648] HybridClock initialized: now 1790281234766159 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T20:20:34Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1344 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:34 dist-test-slave-m1qq krb5kdc[27298](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:34 dist-test-slave-m1qq krb5kdc[27298](info): set up 2 sockets
Sep 24 20:20:34 dist-test-slave-m1qq krb5kdc[27298](info): commencing operation
krb5kdc: starting...
W20260924 20:20:35.869861 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:35 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:36.000169 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35253
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:36.114037 27311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:36.114348 27311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:36.114418 27311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:36.118716 27311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:36.118784 27311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:36.118804 27311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:36.118822 27311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:36.118839 27311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:36.124439 27311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:36.125558 27311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:36.126609 27311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:36.134883 27316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.134863 27317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.135090 27311 server_base.cc:1034] running on GCE node
W20260924 20:20:36.134884 27319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.135715 27311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:36.136737 27311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:36.137935 27311 hybrid_clock.cc:648] HybridClock initialized: now 1790281236137917 us; error 30 us; skew 500 ppm
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:36.140800 27311 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:36.141649 27311 webserver.cc:460] Webserver started at http://127.18.172.126:32857/ using document root <none> and password file <none>
I20260924 20:20:36.142177 27311 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:36.142236 27311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:36.142402 27311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:36.144541 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "2b33d51bb0be438582ae0c1b3413b220"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "28a98514969009278a6a92354aba2da9"
server_key_iv: "24d7ed233bd36fad9f9ba2e62dd9feb3"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.145081 27311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "2b33d51bb0be438582ae0c1b3413b220"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "28a98514969009278a6a92354aba2da9"
server_key_iv: "24d7ed233bd36fad9f9ba2e62dd9feb3"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.148976 27311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 20:20:36.151490 27326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.152858 27311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:36.153008 27311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "2b33d51bb0be438582ae0c1b3413b220"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "28a98514969009278a6a92354aba2da9"
server_key_iv: "24d7ed233bd36fad9f9ba2e62dd9feb3"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.153107 27311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:36.182250 27311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:36.185451 27311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:36.185606 27311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:36.192619 27311 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:35253
I20260924 20:20:36.192646 27378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:35253 every 8 connection(s)
I20260924 20:20:36.193552 27311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:36.196373 27379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:36.196849 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27311
I20260924 20:20:36.197007 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:36.197302 19121 external_mini_cluster.cc:1428] Setting key 0283af3ebcba230da040b81f60900783
I20260924 20:20:36.202220 27379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Bootstrap starting.
I20260924 20:20:36.204964 27379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:36.205916 27379 log.cc:824] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:36.207958 27379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: No bootstrap required, opened a new log
I20260924 20:20:36.211548 27379 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:36.211747 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.198718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34930 (local address 127.18.172.126:35253)
0924 20:20:36.199186 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:36.199200 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:36.199234 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:20:36.199889 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:36.199907 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:36.200229 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:36.200567 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:36.201346 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.202311 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.203568 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.204034 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.204154 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.207118 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:36.207124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:36.207138 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:36.207163 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:36.208921 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.209383 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.209394 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.209399 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.209511 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.209791 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.209795 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.209797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.210187 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:36.210413 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:36.210674 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 20:20:36.210951 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":115,"threads_started":1}
I20260924 20:20:36.211800 27379 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:36.211863 27379 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b33d51bb0be438582ae0c1b3413b220, State: Initialized, Role: FOLLOWER
I20260924 20:20:36.212375 27379 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:36.212544 27379 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:36.212604 27379 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:36.212694 27379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:36.213606 27379 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:36.213976 27379 leader_election.cc:302] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b33d51bb0be438582ae0c1b3413b220; no voters:
I20260924 20:20:36.214277 27379 leader_election.cc:288] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:36.214444 27384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:36.214767 27384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 LEADER]: Becoming Leader. State: Replica: 2b33d51bb0be438582ae0c1b3413b220, State: Running, Role: LEADER
I20260924 20:20:36.215123 27384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:36.215571 27379 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:36.216830 27385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b33d51bb0be438582ae0c1b3413b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } } }
I20260924 20:20:36.216828 27386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b33d51bb0be438582ae0c1b3413b220. Latest consensus state: current_term: 1 leader_uuid: "2b33d51bb0be438582ae0c1b3413b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } } }
I20260924 20:20:36.217006 27385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:36.217056 27386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:36.217880 27395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:36.220669 27395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:36.225834 27395 catalog_manager.cc:1347] Generated new cluster ID: 5974d27784204eb595a573c002bee151
I20260924 20:20:36.225898 27395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:36.244777 27395 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:36.246217 27395 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:36.255512 27395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Generated new TSK 0
I20260924 20:20:36.256412 27395 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:36.298084 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:36.409216 27405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:36.409500 27405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:36.409545 27405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:36.413723 27405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:36.413794 27405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:36.413856 27405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:36.419474 27405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:36.420595 27405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:36.421669 27405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:36.432051 27413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.432075 27410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.432029 27411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.432051 27405 server_base.cc:1034] running on GCE node
I20260924 20:20:36.432754 27405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:36.433425 27405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:36.434625 27405 hybrid_clock.cc:648] HybridClock initialized: now 1790281236434606 us; error 29 us; skew 500 ppm
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:36.437426 27405 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:36.438256 27405 webserver.cc:460] Webserver started at http://127.18.172.65:40771/ using document root <none> and password file <none>
I20260924 20:20:36.438822 27405 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:36.438882 27405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:36.439134 27405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:36.441234 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "c3fe0a26f1dc4b298afd6cb37618c79d"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "74d2c9db9ecde3a09352eb9e89fb10a4"
server_key_iv: "e19b447a3e6ceb7e82863eb7106ad5d7"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.441790 27405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "c3fe0a26f1dc4b298afd6cb37618c79d"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "74d2c9db9ecde3a09352eb9e89fb10a4"
server_key_iv: "e19b447a3e6ceb7e82863eb7106ad5d7"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.445500 27405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 20:20:36.447947 27420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.449131 27405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:36.449247 27405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "c3fe0a26f1dc4b298afd6cb37618c79d"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "74d2c9db9ecde3a09352eb9e89fb10a4"
server_key_iv: "e19b447a3e6ceb7e82863eb7106ad5d7"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.449350 27405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:36.462812 27405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:36.466070 27405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:36.466274 27405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:36.466871 27405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:36.467713 27405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:36.467757 27405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.467823 27405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:36.467852 27405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.477293 27405 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46879
I20260924 20:20:36.477312 27533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46879 every 8 connection(s)
I20260924 20:20:36.478228 27405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:36.484699 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27405
I20260924 20:20:36.484866 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:36.485179 19121 external_mini_cluster.cc:1428] Setting key 5ef8e3f1b4e7c98ab978c1b4a3d13a8e
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:36.492538 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.480073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:39653 (local address 127.18.172.126:35253)
0924 20:20:36.480217 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:36.480220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:36.481299 (+ 1079us) server_negotiation.cc:407] Connection header received
0924 20:20:36.482232 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:36.482236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:36.482292 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:36.482394 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:36.483623 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.484158 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.485565 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.485859 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.485962 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.488859 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:36.488863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:36.488865 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:36.488889 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:36.490293 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.490868 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.490874 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.490877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.490967 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.491406 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.491411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.491412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.491612 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:36.491720 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:36.492244 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 20:20:36.492372 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:20:36.493369 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.480493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:39653)
0924 20:20:36.481148 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:36.481190 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:36.482026 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:36.482570 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:36.482577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:36.482923 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:36.483483 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.483497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.484314 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.484317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.485443 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.485453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.486031 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.486060 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.486220 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.486937 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:36.486954 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:36.488683 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:36.490419 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.490425 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.490441 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.490767 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.491125 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.491128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.491130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.491308 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.491720 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:36.491729 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:36.491991 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 20:20:36.492179 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 20:20:36.492529 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:36.494645 27534 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35253
I20260924 20:20:36.494906 27534 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:36.495515 27534 heartbeater.cc:511] Master 127.18.172.126:35253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:36.497139 27343 ts_manager.cc:194] Registered new tserver with Master: c3fe0a26f1dc4b298afd6cb37618c79d (127.18.172.65:46879)
I20260924 20:20:36.498314 27343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:39653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:36.514518 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:36.625342 27539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:36.625653 27539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:36.625730 27539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:36.629758 27539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:36.629844 27539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:36.629942 27539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:36.635160 27539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:36.636221 27539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:36.637276 27539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:36.647585 27544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.647585 27545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.647636 27539 server_base.cc:1034] running on GCE node
W20260924 20:20:36.647585 27547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.648361 27539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:36.648962 27539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:36.650132 27539 hybrid_clock.cc:648] HybridClock initialized: now 1790281236650114 us; error 28 us; skew 500 ppm
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:36.653249 27539 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:36.654067 27539 webserver.cc:460] Webserver started at http://127.18.172.66:44221/ using document root <none> and password file <none>
I20260924 20:20:36.654587 27539 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:36.654641 27539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:36.654799 27539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:36.657044 27539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "9651c2d163ba4eb096536dbb680b52e3"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "8e3bb76b8a68e70e53a5ddd40cac86dd"
server_key_iv: "1a10dd97a4862539d931981cf49ad636"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.657569 27539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "9651c2d163ba4eb096536dbb680b52e3"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "8e3bb76b8a68e70e53a5ddd40cac86dd"
server_key_iv: "1a10dd97a4862539d931981cf49ad636"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.661540 27539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:20:36.664024 27554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.665272 27539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:36.665390 27539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "9651c2d163ba4eb096536dbb680b52e3"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "8e3bb76b8a68e70e53a5ddd40cac86dd"
server_key_iv: "1a10dd97a4862539d931981cf49ad636"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.665521 27539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:36.683753 27539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:36.686738 27539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:36.686939 27539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:36.687611 27539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:36.688621 27539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:36.688676 27539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.688726 27539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:36.688755 27539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.699885 27539 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36219
I20260924 20:20:36.699941 27667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36219 every 8 connection(s)
I20260924 20:20:36.701041 27539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:36.711318 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27539
I20260924 20:20:36.711437 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:36.711741 19121 external_mini_cluster.cc:1428] Setting key a4119d41a042cd24798ff7fe2686acf7
I20260924 20:20:36.715268 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.703059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57999 (local address 127.18.172.126:35253)
0924 20:20:36.703187 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:36.703190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:36.704321 (+ 1131us) server_negotiation.cc:407] Connection header received
0924 20:20:36.705268 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:36.705271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:36.705331 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:36.705439 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:36.706609 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.707322 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.708447 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.708683 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.708792 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.711285 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:36.711291 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:36.711298 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:36.711331 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:36.712895 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.713516 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.713519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.713520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.713596 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.713989 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.713991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.713992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.714160 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:36.714234 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:36.714876 (+ 642us) server_negotiation.cc:299] Negotiation successful
0924 20:20:36.715047 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:20:36.716355 27670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.703351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:57999)
0924 20:20:36.704158 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:36.704200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:36.705048 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:36.705614 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:36.705621 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:36.705976 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:36.706465 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.706478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.707446 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.707449 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.708322 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.708331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.708830 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.708834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.709001 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.709551 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:36.709567 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:36.711092 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:36.713014 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.713020 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.713038 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.713403 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.713688 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.713693 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.713697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.713895 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.714259 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:36.714269 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:36.714615 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 20:20:36.714834 (+ 219us) client_negotiation.cc:240] Negotiation successful
0924 20:20:36.715251 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":114,"threads_started":1}
I20260924 20:20:36.717669 27668 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35253
I20260924 20:20:36.717927 27668 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:36.718559 27668 heartbeater.cc:511] Master 127.18.172.126:35253 requested a full tablet report, sending...
I20260924 20:20:36.719762 27343 ts_manager.cc:194] Registered new tserver with Master: 9651c2d163ba4eb096536dbb680b52e3 (127.18.172.66:36219)
I20260924 20:20:36.720358 27343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:57999
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:36.742782 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:36.855809 27673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:36.856117 27673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:36.856199 27673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:36.860451 27673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:36.860540 27673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:36.860635 27673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:36.865942 27673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:36.867149 27673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:36.868256 27673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:36.879935 27678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.879967 27681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:36.879942 27679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:36.881965 27673 server_base.cc:1034] running on GCE node
I20260924 20:20:36.882660 27673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:36.883677 27673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:36.884895 27673 hybrid_clock.cc:648] HybridClock initialized: now 1790281236884874 us; error 33 us; skew 500 ppm
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:36.887907 27673 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:36.888895 27673 webserver.cc:460] Webserver started at http://127.18.172.67:46291/ using document root <none> and password file <none>
I20260924 20:20:36.889480 27673 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:36.889572 27673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:36.889804 27673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:36.891932 27673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "1a4b0a4021d54020a164d2af09653ea6"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "0b9f72bb8bc084ddcdc54ac81d79a14c"
server_key_iv: "fa3a21296817c440b28557b17dedccaf"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.892529 27673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "1a4b0a4021d54020a164d2af09653ea6"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "0b9f72bb8bc084ddcdc54ac81d79a14c"
server_key_iv: "fa3a21296817c440b28557b17dedccaf"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.896240 27673 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:36.898697 27688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.899904 27673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:36.900036 27673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "1a4b0a4021d54020a164d2af09653ea6"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "0b9f72bb8bc084ddcdc54ac81d79a14c"
server_key_iv: "fa3a21296817c440b28557b17dedccaf"
server_key_version: "encryptionkey@0"
I20260924 20:20:36.900144 27673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:36.917706 27673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:36.921217 27673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:36.921443 27673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:36.922060 27673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:36.922885 27673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:36.922955 27673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.923044 27673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:36.923090 27673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:36.932504 27673 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:45277
I20260924 20:20:36.932524 27801 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:45277 every 8 connection(s)
I20260924 20:20:36.933393 27673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:36.939343 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27673
I20260924 20:20:36.939570 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:36.939894 19121 external_mini_cluster.cc:1428] Setting key 21b55891a1eaaef7e7ef60e237538b66
Sep 24 20:20:36 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281236, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:36.947461 27382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.935320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:32815 (local address 127.18.172.126:35253)
0924 20:20:36.935499 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:36.935504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:36.936556 (+ 1052us) server_negotiation.cc:407] Connection header received
0924 20:20:36.937400 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:36.937403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:36.937449 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:36.937523 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:36.938807 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.939650 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.941147 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.941439 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.941568 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.944306 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:36.944309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:36.944311 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:36.944335 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:36.945583 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.946043 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.946046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.946047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.946120 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:36.946406 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:36.946408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:36.946409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:36.946613 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:36.946713 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:36.947242 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 20:20:36.947339 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:20:36.948352 27804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:36.935725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:32815)
0924 20:20:36.936384 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:36.936452 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:36.937205 (+ 753us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:36.937687 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:36.937694 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:36.938110 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:36.938673 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.938687 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.939774 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.939778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.940987 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:36.940997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:36.941668 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:36.941673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:36.941889 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:36.942599 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:36.942616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:36.944154 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:36.945685 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.945690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.945703 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.945960 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.946197 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:36.946199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:36.946200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:36.946324 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:36.946701 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:36.946707 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:36.946968 (+ 261us) client_negotiation.cc:769] Sending connection context
0924 20:20:36.947195 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 20:20:36.947503 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":116,"threads_started":1}
I20260924 20:20:36.949460 27802 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35253
I20260924 20:20:36.949661 27802 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:36.950086 27802 heartbeater.cc:511] Master 127.18.172.126:35253 requested a full tablet report, sending...
I20260924 20:20:36.951088 27343 ts_manager.cc:194] Registered new tserver with Master: 1a4b0a4021d54020a164d2af09653ea6 (127.18.172.67:45277)
I20260924 20:20:36.951620 27343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:32815
I20260924 20:20:36.954128 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
Entry for principal oryx/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab.
WARNING: no policy specified for oryx/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
Entry for principal oryx/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab.
WARNING: no policy specified for oryx/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
Entry for principal oryx/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab.
I20260924 20:20:37.030432 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27405
I20260924 20:20:37.035908 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27539
I20260924 20:20:37.041097 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27673
I20260924 20:20:37.046056 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27311
I20260924 20:20:37.052007 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35253
--webserver_interface=127.18.172.126
--webserver_port=32857
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:37.166019 27811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:37.166304 27811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:37.166347 27811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:37.170665 27811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:37.170732 27811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:37.170751 27811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:37.170768 27811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:37.170785 27811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:37.176394 27811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=32857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:37.177578 27811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:37.178568 27811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:37.187451 27819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.187474 27817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.187486 27816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.187651 27811 server_base.cc:1034] running on GCE node
I20260924 20:20:37.188416 27811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:37.189569 27811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:37.190754 27811 hybrid_clock.cc:648] HybridClock initialized: now 1790281237190734 us; error 31 us; skew 500 ppm
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281237, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:37.193600 27811 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:37.194427 27811 webserver.cc:460] Webserver started at http://127.18.172.126:32857/ using document root <none> and password file <none>
I20260924 20:20:37.194953 27811 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:37.195096 27811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:37.198863 27811 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:37.201002 27826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.202220 27811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:37.202344 27811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "2b33d51bb0be438582ae0c1b3413b220"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "28a98514969009278a6a92354aba2da9"
server_key_iv: "24d7ed233bd36fad9f9ba2e62dd9feb3"
server_key_version: "encryptionkey@0"
I20260924 20:20:37.202880 27811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:37.223618 27811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:37.226625 27811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:37.226810 27811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:37.234282 27811 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:35253
I20260924 20:20:37.234373 27878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:35253 every 8 connection(s)
I20260924 20:20:37.235487 27811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:37.239078 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27811
I20260924 20:20:37.239589 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:46879
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:35253
--webserver_port=40771
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:20:37.240162 27879 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:20:37.241902 27879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Bootstrap starting.
I20260924 20:20:37.259107 27879 log.cc:824] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:37.265754 27879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:20:37.266175 27879 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Bootstrap complete.
I20260924 20:20:37.270581 27879 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:37.271087 27879 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b33d51bb0be438582ae0c1b3413b220, State: Initialized, Role: FOLLOWER
I20260924 20:20:37.271646 27879 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:37.271792 27879 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:37.271881 27879 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:37.272003 27879 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:20:37.274121 27879 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:37.274473 27879 leader_election.cc:302] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b33d51bb0be438582ae0c1b3413b220; no voters:
I20260924 20:20:37.274813 27879 leader_election.cc:288] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:20:37.274901 27883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:20:37.275138 27883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [term 2 LEADER]: Becoming Leader. State: Replica: 2b33d51bb0be438582ae0c1b3413b220, State: Running, Role: LEADER
I20260924 20:20:37.275439 27883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } }
I20260924 20:20:37.276129 27879 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:37.276425 27885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b33d51bb0be438582ae0c1b3413b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } } }
I20260924 20:20:37.276557 27885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:37.276908 27887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:37.277005 27884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b33d51bb0be438582ae0c1b3413b220. Latest consensus state: current_term: 2 leader_uuid: "2b33d51bb0be438582ae0c1b3413b220" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b33d51bb0be438582ae0c1b3413b220" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35253 } } }
I20260924 20:20:37.277112 27884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:37.280174 27887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:37.280789 27887 catalog_manager.cc:1259] Loaded cluster ID: 5974d27784204eb595a573c002bee151
I20260924 20:20:37.280848 27887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:37.283203 27887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:37.283872 27887 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b33d51bb0be438582ae0c1b3413b220: Loaded TSK: 0
I20260924 20:20:37.285045 27887 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:20:37.359105 27881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:37.359423 27881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:37.359514 27881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:37.363660 27881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:37.363757 27881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:37.363853 27881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:37.369248 27881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.65.keytab
--principal=oryx/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:46879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=40771
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:37.370301 27881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:37.371366 27881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:37.382447 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.382493 27881 server_base.cc:1034] running on GCE node
W20260924 20:20:37.382431 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.382423 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.383250 27881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:37.384083 27881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:37.385283 27881 hybrid_clock.cc:648] HybridClock initialized: now 1790281237385245 us; error 46 us; skew 500 ppm
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281237, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:37.388798 27881 init.cc:377] Logged in from keytab as oryx/127.18.172.65@KRBTEST.COM (short username oryx)
I20260924 20:20:37.389932 27881 webserver.cc:460] Webserver started at http://127.18.172.65:40771/ using document root <none> and password file <none>
I20260924 20:20:37.390658 27881 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:37.390766 27881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:37.395960 27881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:37.398492 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.400063 27881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:20:37.400180 27881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "c3fe0a26f1dc4b298afd6cb37618c79d"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "74d2c9db9ecde3a09352eb9e89fb10a4"
server_key_iv: "e19b447a3e6ceb7e82863eb7106ad5d7"
server_key_version: "encryptionkey@0"
I20260924 20:20:37.400874 27881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:37.430214 27881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:37.433955 27881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:37.434182 27881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:37.434890 27881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:37.435868 27881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:37.435922 27881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.435971 27881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:37.436000 27881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.445961 27881 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46879
I20260924 20:20:37.445988 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46879 every 8 connection(s)
I20260924 20:20:37.447364 27881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:37.457466 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 27881
I20260924 20:20:37.457918 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:36219
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:35253
--webserver_port=44221
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.461037 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.449796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:50181)
0924 20:20:37.450516 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.450558 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.451405 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.452486 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.452492 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.452848 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.453356 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.453371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.454732 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.454735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.455683 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.455692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.456390 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.456393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.456537 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.457153 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.457169 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.460066 (+ 2897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":138,"threads_started":1}
W20260924 20:20:37.461064 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.449610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:50181 (local address 127.18.172.126:35253)
0924 20:20:37.450060 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.450084 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.450682 (+ 598us) server_negotiation.cc:407] Connection header received
0924 20:20:37.451777 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.451789 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.452088 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.452441 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.453530 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.454596 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.455792 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.456254 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.456443 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.460269 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:50181: BlockingRecv error: recv got EOF from 127.18.172.65:50181 (error 108)
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":96,"threads_started":1}
W20260924 20:20:37.461536 28030 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:35253 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.172.126:35253: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.467792 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.462003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:54429)
0924 20:20:37.462147 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.462164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.462245 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.462504 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.462510 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.462700 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.462841 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.462851 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.463636 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.463642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.464675 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.464687 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.465116 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.465122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.465246 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.465705 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.465724 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.467652 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 20:20:37.467878 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.462097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54429 (local address 127.18.172.126:35253)
0924 20:20:37.462207 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.462210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.462223 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:37.462339 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.462341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.462381 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.462450 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.462974 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.463526 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.464793 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.465005 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.465081 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.467767 (+ 2686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:54429: BlockingRecv error: recv got EOF from 127.18.172.65:54429 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.473856 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.468355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:56141)
0924 20:20:37.468585 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.468596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.468652 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.468949 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.468952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.469111 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.469201 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.469206 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.470300 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.470303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.470807 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.470814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.471360 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.471362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.471422 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.471887 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.471899 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.473761 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167}
W20260924 20:20:37.474093 28030 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:20:37.474208 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.468469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:56141 (local address 127.18.172.126:35253)
0924 20:20:37.468607 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.468612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.468630 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:20:37.468736 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.468742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.468797 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.468891 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.469308 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.470183 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.470914 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.471264 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.471361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.473911 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:56141: BlockingRecv error: recv got EOF from 127.18.172.65:56141 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 20:20:37.572433 28034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:37.572713 28034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:37.572757 28034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:37.577122 28034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:37.577210 28034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:37.577315 28034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:37.582677 28034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.66.keytab
--principal=oryx/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:36219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=44221
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:37.583783 28034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:37.584770 28034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:37.594111 28039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.594129 28042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.594239 28034 server_base.cc:1034] running on GCE node
W20260924 20:20:37.594411 28040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.594800 28034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:37.595522 28034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:37.596709 28034 hybrid_clock.cc:648] HybridClock initialized: now 1790281237596687 us; error 31 us; skew 500 ppm
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281237, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:37.599515 28034 init.cc:377] Logged in from keytab as oryx/127.18.172.66@KRBTEST.COM (short username oryx)
I20260924 20:20:37.600346 28034 webserver.cc:460] Webserver started at http://127.18.172.66:44221/ using document root <none> and password file <none>
I20260924 20:20:37.600898 28034 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:37.601001 28034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:37.604813 28034 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 20:20:37.607213 28049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.608629 28034 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:37.608769 28034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "9651c2d163ba4eb096536dbb680b52e3"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "8e3bb76b8a68e70e53a5ddd40cac86dd"
server_key_iv: "1a10dd97a4862539d931981cf49ad636"
server_key_version: "encryptionkey@0"
I20260924 20:20:37.609339 28034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:37.622737 28034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:37.626151 28034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:37.626353 28034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:37.626941 28034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:37.627786 28034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:37.627866 28034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.627942 28034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:37.627990 28034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.637219 28034 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36219
I20260924 20:20:37.637300 28163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36219 every 8 connection(s)
I20260924 20:20:37.638259 28034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:37.645908 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28034
I20260924 20:20:37.646412 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:45277
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:35253
--webserver_port=46291
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:20:37.650214 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.640197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:44209 (local address 127.18.172.126:35253)
0924 20:20:37.640338 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.640342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.641766 (+ 1424us) server_negotiation.cc:407] Connection header received
0924 20:20:37.642720 (+ 954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.642723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.642794 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.642876 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.644093 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.644623 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.645899 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.646218 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.646314 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.650032 (+ 3718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:44209: BlockingRecv error: recv got EOF from 127.18.172.66:44209 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:20:37.650707 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.640581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:44209)
0924 20:20:37.641588 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.641632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.642507 (+ 875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.643094 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.643100 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.643471 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.643950 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.643963 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.644753 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.644757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.645726 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.645738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.646388 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.646392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.646620 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.647324 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.647351 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.649815 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":790,"thread_start_us":534,"threads_started":1}
W20260924 20:20:37.651283 28164 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:35253 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.172.126:35253: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.657586 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.651888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:46861)
0924 20:20:37.651993 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.652010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.652128 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.652409 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.652412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.652593 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.652700 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.652706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.653529 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.653535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.654147 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.654156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.654657 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.654661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.654779 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.655343 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.655355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.657422 (+ 2067us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 20:20:37.657678 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.652005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:46861 (local address 127.18.172.126:35253)
0924 20:20:37.652147 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.652154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.652173 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:37.652224 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.652227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.652283 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.652391 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.652840 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.653411 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.654289 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.654530 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.654653 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.657548 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:46861: BlockingRecv error: recv got EOF from 127.18.172.66:46861 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.663952 28166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.658331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:56663)
0924 20:20:37.658456 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.658471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.658546 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.658764 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.658767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.658928 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.659046 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.659053 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.659880 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.659883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.660741 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.660751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.661192 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.661196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.661277 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.661745 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.661759 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.663828 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 20:20:37.664116 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.658446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:56663 (local address 127.18.172.126:35253)
0924 20:20:37.658554 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.658558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.658571 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:37.658624 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.658626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.658662 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.658729 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.659161 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.659754 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.660867 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.661088 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.661173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.663967 (+ 2794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:56663: BlockingRecv error: recv got EOF from 127.18.172.66:56663 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260924 20:20:37.664158 28164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:20:37.762099 28167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:37.762367 28167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:37.762409 28167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:37.766425 28167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:37.766498 28167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:37.766558 28167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:37.771718 28167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/krb5kdc/oryx_127.18.172.67.keytab
--principal=oryx/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:45277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=46291
--tserver_master_addrs=127.18.172.126:35253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:37.772711 28167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:37.773672 28167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:37.783784 28173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.783761 28172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:37.783761 28175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:37.783969 28167 server_base.cc:1034] running on GCE node
I20260924 20:20:37.784555 28167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:37.785234 28167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:37.786422 28167 hybrid_clock.cc:648] HybridClock initialized: now 1790281237786391 us; error 42 us; skew 500 ppm
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281237, etypes {rep=17 tkt=17 ses=17}, oryx/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:37.789345 28167 init.cc:377] Logged in from keytab as oryx/127.18.172.67@KRBTEST.COM (short username oryx)
I20260924 20:20:37.790333 28167 webserver.cc:460] Webserver started at http://127.18.172.67:46291/ using document root <none> and password file <none>
I20260924 20:20:37.790897 28167 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:37.791006 28167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:37.794891 28167 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:37.796852 28183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.798039 28167 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:37.798175 28167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "1a4b0a4021d54020a164d2af09653ea6"
format_stamp: "Formatted at 2026-09-24 20:20:36 on dist-test-slave-m1qq"
server_key: "0b9f72bb8bc084ddcdc54ac81d79a14c"
server_key_iv: "fa3a21296817c440b28557b17dedccaf"
server_key_version: "encryptionkey@0"
I20260924 20:20:37.798700 28167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:37.832844 28167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:37.835846 28167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:37.836062 28167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:37.836673 28167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:37.837513 28167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:37.837584 28167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.837651 28167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:37.837700 28167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:37.847857 28167 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:45277
I20260924 20:20:37.847889 28296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:45277 every 8 connection(s)
I20260924 20:20:37.848881 28167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:37.853895 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28167
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:20:37.860229 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.850736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:59175 (local address 127.18.172.126:35253)
0924 20:20:37.850882 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.850886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.852066 (+ 1180us) server_negotiation.cc:407] Connection header received
0924 20:20:37.852940 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.852943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.852997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.853073 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.854526 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.855280 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.856402 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.856630 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.856742 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.860110 (+ 3368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:59175: BlockingRecv error: recv got EOF from 127.18.172.67:59175 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 20:20:37.860958 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.851270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:59175)
0924 20:20:37.851915 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.851956 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.852739 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.853241 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.853247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.853639 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.854372 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.854389 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.855399 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.855403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.856279 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.856288 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.856790 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.856794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.856970 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.857670 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.857695 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.860014 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":107,"threads_started":1}
W20260924 20:20:37.861503 28297 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:35253 (0 consecutive failures): Not authorized: Failed to ping master at 127.18.172.126:35253: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
I20260924 20:20:37.863989 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.854932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.18.172.126:35253)
0924 20:20:37.855151 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.855154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.855166 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:37.855199 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.855200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.855236 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.855442 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.856163 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.856683 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.857769 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.858001 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.858108 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.859581 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:37.859586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:37.859600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:37.859625 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:37.861496 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:37.861992 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:37.862152 (+ 160us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:37.862158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:37.862301 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:37.862605 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:37.862607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:37.862609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:37.863064 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:37.863242 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:37.863604 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 20:20:37.863868 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":95,"threads_started":1}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.867430 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.861967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:46205)
0924 20:20:37.862083 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.862097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.862170 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.862406 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.862409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.862566 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.862688 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.862693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.863615 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.863620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.864319 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.864329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.864767 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.864771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.864868 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.865359 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.865370 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.867342 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:20:37.867630 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.862074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:46205 (local address 127.18.172.126:35253)
0924 20:20:37.862187 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.862194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.862210 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:37.862245 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.862247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.862282 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.862370 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.862787 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.863504 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.864465 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.864662 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.864739 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.867518 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:46205: BlockingRecv error: recv got EOF from 127.18.172.67:46205 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:37 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:37.872256 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:37.867906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:56343)
0924 20:20:37.868020 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:37.868035 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:37.868098 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:37.868318 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:37.868323 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:37.868443 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:37.868507 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.868510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.869126 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.869128 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.869636 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:37.869642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.870078 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.870081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:37.870133 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.870494 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:37.870504 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:37.872164 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:20:37.872430 28032 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:37.868029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:56343 (local address 127.18.172.126:35253)
0924 20:20:37.868159 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:37.868163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:37.868182 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:37.868210 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:37.868212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:37.868243 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:37.868304 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:37.868592 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.869041 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.869740 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:37.869987 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:37.870064 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:37.872310 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:56343: BlockingRecv error: recv got EOF from 127.18.172.67:56343 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 20:20:37.872498 28297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:38.481149 28302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:38.474813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:54487)
0924 20:20:38.475124 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:38.475137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:38.475230 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:38.475549 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:38.475550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:38.475761 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:38.475888 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.475895 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.476846 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.476850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.477471 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.477480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.478191 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.478193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.478269 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.478861 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:38.478879 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:38.480975 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":107,"threads_started":1}
W20260924 20:20:38.481247 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:38.474879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54487 (local address 127.18.172.126:35253)
0924 20:20:38.475159 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:38.475162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:38.475180 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:20:38.475361 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:38.475363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:38.475414 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:38.475502 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:38.476005 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.476649 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.477631 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.477981 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.478198 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.481108 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:54487: BlockingRecv error: recv got EOF from 127.18.172.65:54487 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:38.670398 28304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:38.664812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:52757)
0924 20:20:38.665056 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:38.665073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:38.665184 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:38.665526 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:38.665529 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:38.665731 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:38.665842 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.665848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.666625 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.666630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.667234 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.667242 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.667691 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.667695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.667789 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.668324 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:38.668336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:38.670282 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":80,"threads_started":1}
W20260924 20:20:38.670624 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:38.664930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52757 (local address 127.18.172.126:35253)
0924 20:20:38.665080 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:38.665086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:38.665121 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:38.665304 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:38.665308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:38.665385 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:38.665500 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:38.665989 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.666517 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.667352 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.667578 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.667688 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.670490 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:52757: BlockingRecv error: recv got EOF from 127.18.172.66:52757 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:38 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 20:20:38.881314 28303 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:38.873213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:34775 (local address 127.18.172.126:35253)
0924 20:20:38.873358 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:38.873361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:38.873428 (+ 67us) server_negotiation.cc:407] Connection header received
0924 20:20:38.873594 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:38.873597 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:38.873656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:38.873748 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:38.874469 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.875473 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.876987 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.877370 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.877447 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.881188 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:34775: BlockingRecv error: recv got EOF from 127.18.172.67:34775 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:20:38.881618 28305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:38.873107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:34775)
0924 20:20:38.873363 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:38.873376 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:38.873492 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:38.873811 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:38.873814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:38.874111 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:38.874305 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.874313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.875701 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.875708 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.876798 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:38.876812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:38.877624 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:38.877631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:38.877831 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:38.878424 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:38.878444 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:38.881456 (+ 3012us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":95,"threads_started":1}
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:39.488608 28306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:39.482052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:50857)
0924 20:20:39.482347 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:39.482362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:39.482482 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:39.482735 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:39.482737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:39.482917 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:39.483043 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.483048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.484040 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.484043 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.485187 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.485198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.485641 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.485643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.485707 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.486220 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:39.486232 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:39.488391 (+ 2159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":97,"threads_started":1}
W20260924 20:20:39.488615 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:39.482119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:50857 (local address 127.18.172.126:35253)
0924 20:20:39.482412 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:39.482416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:39.482435 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:20:39.482576 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:39.482578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:39.482623 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:39.482699 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:39.483151 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.483889 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.485313 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.485538 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.485617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.488513 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:50857: BlockingRecv error: recv got EOF from 127.18.172.65:50857 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:39.676546 28308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:39.671234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:50681)
0924 20:20:39.671512 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:39.671525 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:39.671623 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:39.671851 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:39.671853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:39.672042 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:39.672135 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.672140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.672866 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.672868 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.673430 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.673439 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.673825 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.673828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.673886 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.674443 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:39.674476 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:39.676403 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
W20260924 20:20:39.676661 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:39.671292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:50681 (local address 127.18.172.126:35253)
0924 20:20:39.671451 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:39.671456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:39.671597 (+ 141us) server_negotiation.cc:407] Connection header received
0924 20:20:39.671703 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:39.671705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:39.671750 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:39.671818 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:39.672243 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.672766 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.673530 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.673734 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.673807 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.676545 (+ 2738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:50681: BlockingRecv error: recv got EOF from 127.18.172.66:50681 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:39 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:39.892982 28309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:39.883860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:46631)
0924 20:20:39.884184 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:39.884198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:39.884300 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:39.884547 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:39.884549 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:39.884718 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:39.884813 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.884818 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.885715 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.885717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.886300 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:39.886309 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.886784 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.886790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:39.886866 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.890879 (+ 4013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:39.890892 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:39.892825 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":103,"threads_started":1}
W20260924 20:20:39.893067 28307 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:39.884073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:46631 (local address 127.18.172.126:35253)
0924 20:20:39.884217 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:39.884221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:39.884253 (+ 32us) server_negotiation.cc:407] Connection header received
0924 20:20:39.884388 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:39.884390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:39.884437 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:39.884513 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:39.884957 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.885605 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.886429 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:39.886663 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:39.886746 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:39.892974 (+ 6228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:46631: BlockingRecv error: recv got EOF from 127.18.172.67:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:40.495532 28310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:40.489514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:34905)
0924 20:20:40.489809 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:40.489824 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:40.489930 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:40.490186 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:40.490189 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:40.490376 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:40.490512 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.490518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.491401 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.491405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.492406 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.492417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.492835 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.492838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.492900 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.493511 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:40.493524 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:40.495370 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":84,"threads_started":1}
W20260924 20:20:40.495649 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:40.489537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:34905 (local address 127.18.172.126:35253)
0924 20:20:40.489823 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:40.489827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:40.489923 (+ 96us) server_negotiation.cc:407] Connection header received
0924 20:20:40.490019 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:40.490021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:40.490066 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:40.490144 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:40.490614 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.491258 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.492520 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.492738 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.492817 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.495525 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:34905: BlockingRecv error: recv got EOF from 127.18.172.65:34905 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:40.684228 28312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:40.677751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:60303)
0924 20:20:40.678091 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:40.678104 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:40.678216 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:40.678588 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:40.678594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:40.678833 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:40.679028 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.679039 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.680053 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.680057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.680928 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.680937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.681376 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.681379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.681450 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.682017 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:40.682036 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:40.684049 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":110,"threads_started":1}
W20260924 20:20:40.684283 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:40.677814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:60303 (local address 127.18.172.126:35253)
0924 20:20:40.677969 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:40.677974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:40.678181 (+ 207us) server_negotiation.cc:407] Connection header received
0924 20:20:40.678368 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:40.678370 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:40.678448 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:40.678542 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:40.679159 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.679930 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.681032 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.681247 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.681331 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.684172 (+ 2841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:60303: BlockingRecv error: recv got EOF from 127.18.172.66:60303 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:40 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:40.898808 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:40.893746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:47565)
0924 20:20:40.894007 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:40.894024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:40.894119 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:40.894345 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:40.894347 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:40.894528 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:40.894621 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.894627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.895368 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.895371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.895930 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:40.895938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.896311 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.896313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:40.896370 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.896882 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:40.896894 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:40.898677 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260924 20:20:40.898916 28311 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:40.893829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:47565 (local address 127.18.172.126:35253)
0924 20:20:40.893961 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:40.893969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:40.894086 (+ 117us) server_negotiation.cc:407] Connection header received
0924 20:20:40.894194 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:40.894196 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:40.894241 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:40.894312 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:40.894711 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.895283 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.896020 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:40.896223 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:40.896296 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:40.898803 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:47565: BlockingRecv error: recv got EOF from 127.18.172.67:47565 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:41.502539 28314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:41.496443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:42847)
0924 20:20:41.496717 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:41.496731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:41.496821 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:41.497064 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:41.497066 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:41.497251 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:41.497405 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.497411 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.498374 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.498377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.499292 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.499304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.499868 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.499872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.499941 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.500497 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:41.500509 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:41.502352 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260924 20:20:41.502602 28315 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:41.496595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:42847 (local address 127.18.172.126:35253)
0924 20:20:41.496834 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:41.496837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:41.496852 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:20:41.496897 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:41.496900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:41.496945 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:41.497034 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:41.497538 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.498246 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.499411 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.499631 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.499721 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.502483 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:42847: BlockingRecv error: recv got EOF from 127.18.172.65:42847 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":92,"threads_started":1}
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:41.691175 28316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:41.685107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:55607)
0924 20:20:41.685377 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:41.685395 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:41.685534 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:41.685808 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:41.685811 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:41.686040 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:41.686145 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.686151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.687104 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.687106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.687802 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.687817 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.688528 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.688531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.688611 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.689254 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:41.689266 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:41.691060 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
W20260924 20:20:41.691325 28315 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:41.685206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:55607 (local address 127.18.172.126:35253)
0924 20:20:41.685339 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:41.685343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:41.685487 (+ 144us) server_negotiation.cc:407] Connection header received
0924 20:20:41.685618 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:41.685621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:41.685668 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:41.685747 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:41.686299 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.686918 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.687939 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.688379 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.688493 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.691209 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:55607: BlockingRecv error: recv got EOF from 127.18.172.66:55607 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:41 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:41.905289 28317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:41.899622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:58273)
0924 20:20:41.899918 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:41.899932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:41.900053 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:41.900444 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:41.900447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:41.900673 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:41.900769 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.900775 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.901604 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.901607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.902241 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:41.902250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.902653 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.902655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:41.902714 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.903239 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:41.903251 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:41.905143 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":111,"threads_started":1}
W20260924 20:20:41.905529 28315 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:41.899721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:58273 (local address 127.18.172.126:35253)
0924 20:20:41.899867 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:41.899872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:41.900006 (+ 134us) server_negotiation.cc:407] Connection header received
0924 20:20:41.900139 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:41.900141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:41.900206 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:41.900290 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:41.900906 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.901497 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.902334 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:41.902562 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:41.902642 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:41.905286 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:58273: BlockingRecv error: recv got EOF from 127.18.172.67:58273 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:42.509002 28318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:42.503545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:49613)
0924 20:20:42.503824 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:42.503838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:42.503930 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:42.504152 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:42.504154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:42.504343 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:42.504444 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.504450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.505198 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.505202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.505784 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.505792 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.506332 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.506335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.506435 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.506930 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:42.506941 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:42.508854 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":95,"threads_started":1}
W20260924 20:20:42.509179 28320 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:42.503607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:49613 (local address 127.18.172.126:35253)
0924 20:20:42.503863 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:42.503867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:42.503881 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:42.504016 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:42.504019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:42.504065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:42.504144 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:42.504565 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.505096 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.505913 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.506197 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.506327 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.509016 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:49613: BlockingRecv error: recv got EOF from 127.18.172.65:49613 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":59,"threads_started":1}
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:42.697795 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:42.692238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:39947)
0924 20:20:42.692546 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:42.692559 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:42.692648 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:42.692876 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:42.692879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:42.693060 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:42.693150 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.693155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.693902 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.693906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.694578 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.694587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.695111 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.695114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.695208 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.695761 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:42.695773 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:42.697628 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":98,"threads_started":1}
W20260924 20:20:42.697849 28320 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:42.692356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:39947 (local address 127.18.172.126:35253)
0924 20:20:42.692524 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:42.692528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:42.692621 (+ 93us) server_negotiation.cc:407] Connection header received
0924 20:20:42.692723 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:42.692725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:42.692769 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:42.692845 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:42.693246 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.693773 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.694700 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.694954 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.695101 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.697744 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:39947: BlockingRecv error: recv got EOF from 127.18.172.66:39947 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:42 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:42.911757 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:42.906004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:57739)
0924 20:20:42.906503 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:42.906518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:42.906662 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:42.906977 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:42.906979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:42.907392 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:42.907523 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.907530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.908317 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.908320 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.908904 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:42.908913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.909307 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.909310 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:42.909416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.909866 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:42.909879 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:42.911641 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":111,"threads_started":1}
W20260924 20:20:42.911885 28320 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:42.906091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:57739 (local address 127.18.172.126:35253)
0924 20:20:42.906238 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:42.906243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:42.906602 (+ 359us) server_negotiation.cc:407] Connection header received
0924 20:20:42.906780 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:42.906782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:42.906831 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:42.906903 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:42.907649 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.908199 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.909004 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:42.909217 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:42.909290 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:42.911769 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:57739: BlockingRecv error: recv got EOF from 127.18.172.67:57739 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:43.515569 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:43.509713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:57363)
0924 20:20:43.509968 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:43.509981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:43.510072 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:43.510311 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:43.510313 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:43.510498 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:43.510591 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.510597 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.511396 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.511399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.511978 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.511987 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.512497 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.512501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.512579 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.513106 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:43.513119 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:43.515447 (+ 2328us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":82,"threads_started":1}
W20260924 20:20:43.515722 28324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:43.509760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:57363 (local address 127.18.172.126:35253)
0924 20:20:43.509998 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:43.510001 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:43.510032 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:20:43.510154 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:43.510156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:43.510198 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:43.510275 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:43.510695 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.511271 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.512103 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.512353 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.512432 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.515583 (+ 3151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:57363: BlockingRecv error: recv got EOF from 127.18.172.65:57363 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":90,"threads_started":1}
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:43.704137 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:43.698591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:38331)
0924 20:20:43.698847 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:43.698860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:43.698986 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:43.699236 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:43.699238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:43.699422 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:43.699515 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.699528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.700250 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.700253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.700825 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.700832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.701261 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.701264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.701324 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.701774 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:43.701785 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:43.704011 (+ 2226us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
W20260924 20:20:43.704263 28324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:43.698638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:38331 (local address 127.18.172.126:35253)
0924 20:20:43.698785 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:43.698789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:43.698955 (+ 166us) server_negotiation.cc:407] Connection header received
0924 20:20:43.699108 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:43.699110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:43.699157 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:43.699225 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:43.699621 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.700142 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.700955 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.701170 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.701242 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.704144 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:38331: BlockingRecv error: recv got EOF from 127.18.172.66:38331 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:43 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:43.918617 28326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:43.912661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:42753)
0924 20:20:43.912972 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:43.912988 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:43.913152 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:43.913514 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:43.913517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:43.913738 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:43.913832 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.913838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.914882 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.914885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.915605 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:43.915614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.916039 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.916041 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:43.916104 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.916657 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:43.916668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:43.918488 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":147,"threads_started":1}
W20260924 20:20:43.918942 28324 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:43.912757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:42753 (local address 127.18.172.126:35253)
0924 20:20:43.912892 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:43.912896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:43.913070 (+ 174us) server_negotiation.cc:407] Connection header received
0924 20:20:43.913268 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:43.913271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:43.913329 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:43.913430 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:43.913998 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.914766 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.915710 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:43.915935 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:43.916013 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:43.918648 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:42753: BlockingRecv error: recv got EOF from 127.18.172.67:42753 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:44.522009 28327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:44.516436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:54379)
0924 20:20:44.516775 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:44.516789 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:44.516892 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:44.517130 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:44.517132 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:44.517313 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:44.517405 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.517410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.518117 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.518119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.518709 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.518717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.519181 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.519184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.519247 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.519813 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:44.519831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:44.521891 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":87,"threads_started":1}
W20260924 20:20:44.522156 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:44.516546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54379 (local address 127.18.172.126:35253)
0924 20:20:44.516781 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:44.516785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:44.516865 (+ 80us) server_negotiation.cc:407] Connection header received
0924 20:20:44.516993 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:44.516996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:44.517042 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:44.517118 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:44.517504 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.518025 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.518821 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.519078 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.519154 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.522027 (+ 2873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:54379: BlockingRecv error: recv got EOF from 127.18.172.65:54379 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":59,"threads_started":1}
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:44.711356 28329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:44.705102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:46487)
0924 20:20:44.705563 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:44.705581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:44.705764 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:44.706045 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:44.706048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:44.706266 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:44.706408 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.706416 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.707723 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.707726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.708302 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.708310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.708760 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.708762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.708824 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.709331 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:44.709342 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:44.711174 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
W20260924 20:20:44.711400 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:44.705152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:46487 (local address 127.18.172.126:35253)
0924 20:20:44.705444 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:44.705452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:44.705710 (+ 258us) server_negotiation.cc:407] Connection header received
0924 20:20:44.705880 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:44.705883 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:44.705939 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:44.706027 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:44.706553 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.707580 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.708429 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.708661 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.708748 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.711292 (+ 2544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:46487: BlockingRecv error: recv got EOF from 127.18.172.66:46487 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:44 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:44.929504 28330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:44.919398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:45919)
0924 20:20:44.919743 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:44.919757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:44.919903 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:44.920177 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:44.920179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:44.920398 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:44.920486 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.920492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.921370 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.921373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.921922 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:44.921930 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.922380 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.922384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:44.922479 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.927374 (+ 4895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:44.927390 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:44.929320 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":87,"threads_started":1}
W20260924 20:20:44.929616 28328 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:44.919484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:45919 (local address 127.18.172.126:35253)
0924 20:20:44.919679 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:44.919685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:44.919849 (+ 164us) server_negotiation.cc:407] Connection header received
0924 20:20:44.920027 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:44.920029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:44.920072 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:44.920151 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:44.920608 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.921148 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.922040 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:44.922269 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:44.922376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:44.929485 (+ 7109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:45919: BlockingRecv error: recv got EOF from 127.18.172.67:45919 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:45.528378 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:45.522932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:42189)
0924 20:20:45.523234 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:45.523247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:45.523369 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:45.523713 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:45.523716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:45.523906 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:45.524004 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.524010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.524833 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.524836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.525406 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.525414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.525802 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.525804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.525866 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.526338 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:45.526348 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:45.528252 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":128,"threads_started":1}
W20260924 20:20:45.528550 28332 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:45.523079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:42189 (local address 127.18.172.126:35253)
0924 20:20:45.523338 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:45.523342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:45.523354 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:45.523507 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:45.523510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:45.523569 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:45.523662 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:45.524128 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.524710 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.525502 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.525715 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.525787 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.528378 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:42189: BlockingRecv error: recv got EOF from 127.18.172.65:42189 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:45.717556 28333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:45.712079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:35919)
0924 20:20:45.712338 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:45.712351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:45.712444 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:45.712733 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:45.712736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:45.712931 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:45.713030 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.713035 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.714111 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.714114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.714693 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.714701 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.715128 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.715131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.715193 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.715643 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:45.715653 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:45.717443 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":98,"threads_started":1}
W20260924 20:20:45.717705 28332 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:45.712138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:35919 (local address 127.18.172.126:35253)
0924 20:20:45.712265 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:45.712269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:45.712413 (+ 144us) server_negotiation.cc:407] Connection header received
0924 20:20:45.712540 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:45.712542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:45.712620 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:45.712700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:45.713158 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.714011 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.714788 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.715043 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.715120 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.717571 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:35919: BlockingRecv error: recv got EOF from 127.18.172.66:35919 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:45 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:45.936054 28334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:45.930548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:51635)
0924 20:20:45.930824 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:45.930837 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:45.930962 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:45.931216 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:45.931218 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:45.931394 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:45.931490 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.931495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.932330 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.932335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.932958 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:45.932967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.933401 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.933405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:45.933479 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.933984 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:45.933996 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:45.935880 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
W20260924 20:20:45.936110 28332 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:45.930622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:51635 (local address 127.18.172.126:35253)
0924 20:20:45.930765 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:45.930770 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:45.930913 (+ 143us) server_negotiation.cc:407] Connection header received
0924 20:20:45.931066 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:45.931069 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:45.931116 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:45.931184 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:45.931630 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.932223 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.933071 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:45.933296 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:45.933406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:45.936003 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:51635: BlockingRecv error: recv got EOF from 127.18.172.67:51635 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:46.535079 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:46.529121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:57183)
0924 20:20:46.529465 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:46.529480 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:46.529575 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:46.529824 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:46.529826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:46.530048 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:46.530147 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.530153 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.530938 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.530942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.531617 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.531626 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.532057 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.532060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.532127 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.532748 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:46.532761 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:46.534860 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":90,"threads_started":1}
W20260924 20:20:46.535137 28336 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:46.529252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:57183 (local address 127.18.172.126:35253)
0924 20:20:46.529509 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:46.529512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:46.529526 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:46.529667 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:46.529669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:46.529712 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:46.529787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:46.530247 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.530796 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.531731 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.531959 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.532038 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.534986 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:57183: BlockingRecv error: recv got EOF from 127.18.172.65:57183 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":59,"threads_started":1}
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:46.723650 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:46.718384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:53045)
0924 20:20:46.718649 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:46.718662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:46.718763 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:46.718987 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:46.718989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:46.719183 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:46.719281 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.719286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.720044 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.720046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.720606 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.720615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.721008 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.721010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.721069 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.721613 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:46.721625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:46.723532 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":98,"threads_started":1}
W20260924 20:20:46.723908 28336 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:46.718453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:53045 (local address 127.18.172.126:35253)
0924 20:20:46.718598 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:46.718602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:46.718731 (+ 129us) server_negotiation.cc:407] Connection header received
0924 20:20:46.718829 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:46.718831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:46.718875 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:46.718954 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:46.719419 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.719947 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.720705 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.720918 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.720991 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.723779 (+ 2788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:53045: BlockingRecv error: recv got EOF from 127.18.172.66:53045 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:46 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:46.942059 28338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:46.936937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:54573)
0924 20:20:46.937202 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:46.937215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:46.937312 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:46.937575 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:46.937577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:46.937758 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:46.937855 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.937860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.938540 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.938542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.939119 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:46.939126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.939519 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.939522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:46.939584 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.940134 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:46.940146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:46.941942 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
W20260924 20:20:46.942219 28336 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:46.936999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:54573 (local address 127.18.172.126:35253)
0924 20:20:46.937141 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:46.937145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:46.937283 (+ 138us) server_negotiation.cc:407] Connection header received
0924 20:20:46.937427 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:46.937428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:46.937472 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:46.937545 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:46.937948 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.938452 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.939209 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:46.939426 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:46.939501 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:46.942064 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:54573: BlockingRecv error: recv got EOF from 127.18.172.67:54573 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:47.541239 28339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:47.535877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:46529)
0924 20:20:47.536149 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:47.536255 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:47.536355 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:47.536564 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:47.536566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:47.536743 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:47.536838 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.536845 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.537618 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.537620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.538183 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.538190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.538645 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.538647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.538718 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.539290 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:47.539301 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:47.541112 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":84,"threads_started":1}
W20260924 20:20:47.541391 28340 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:47.535931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46529 (local address 127.18.172.126:35253)
0924 20:20:47.536204 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:47.536208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:47.536327 (+ 119us) server_negotiation.cc:407] Connection header received
0924 20:20:47.536420 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:47.536422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:47.536468 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:47.536543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:47.536953 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.537521 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.538283 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.538545 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.538622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.541241 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:46529: BlockingRecv error: recv got EOF from 127.18.172.65:46529 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":89,"threads_started":1}
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:47.729851 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:47.724455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:40567)
0924 20:20:47.724724 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:47.724739 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:47.724857 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:47.725123 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:47.725126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:47.725333 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:47.725424 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.725430 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.726194 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.726199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.726810 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.726818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.727330 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.727333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.727411 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.727924 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:47.727935 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:47.729720 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
W20260924 20:20:47.730026 28340 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:47.724537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:40567 (local address 127.18.172.126:35253)
0924 20:20:47.724696 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:47.724701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:47.724806 (+ 105us) server_negotiation.cc:407] Connection header received
0924 20:20:47.724967 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:47.724970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:47.725018 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:47.725101 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:47.725540 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.726077 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.726937 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.727212 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.727314 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.729885 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:40567: BlockingRecv error: recv got EOF from 127.18.172.66:40567 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:47 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:47.948074 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:47.942804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:43233)
0924 20:20:47.943132 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:47.943152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:47.943256 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:47.943527 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:47.943529 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:47.943701 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:47.943792 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.943797 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.944513 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.944515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.945082 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:47.945089 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.945493 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.945496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:47.945555 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.946007 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:47.946018 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:47.947947 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
W20260924 20:20:47.948181 28340 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:47.942971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:43233 (local address 127.18.172.126:35253)
0924 20:20:47.943159 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:47.943166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:47.943202 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:20:47.943351 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:47.943353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:47.943427 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:47.943493 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:47.943901 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.944422 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.945175 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:47.945400 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:47.945472 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:47.948073 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:43233: BlockingRecv error: recv got EOF from 127.18.172.67:43233 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:48.547663 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:48.542148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:37287)
0924 20:20:48.542421 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:48.542439 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:48.542546 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:48.542788 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:48.542790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:48.543056 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:48.543164 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.543170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.543945 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.543951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.544740 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.544751 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.545183 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.545186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.545269 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.545750 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:48.545761 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:48.547542 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":68,"threads_started":1}
W20260924 20:20:48.547798 28344 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:48.542295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:37287 (local address 127.18.172.126:35253)
0924 20:20:48.542521 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:48.542525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:48.542538 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:48.542627 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:48.542629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:48.542672 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:48.542748 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:48.543268 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.543801 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.544839 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.545071 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.545178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.547671 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:37287: BlockingRecv error: recv got EOF from 127.18.172.65:37287 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:48.736938 28345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:48.730924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:46049)
0924 20:20:48.731236 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:48.731253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:48.731428 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:48.731708 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:48.731710 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:48.731902 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:48.731999 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.732004 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.732742 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.732745 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.733349 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.733359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.733972 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.733975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.734054 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.734672 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:48.734685 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:48.736799 (+ 2114us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":127,"threads_started":1}
W20260924 20:20:48.737202 28344 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:48.730941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:46049 (local address 127.18.172.126:35253)
0924 20:20:48.731130 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:48.731134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:48.731379 (+ 245us) server_negotiation.cc:407] Connection header received
0924 20:20:48.731543 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:48.731546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:48.731601 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:48.731674 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:48.732099 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.732617 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.733483 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.733739 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.733829 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.737056 (+ 3227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:46049: BlockingRecv error: recv got EOF from 127.18.172.66:46049 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:48 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:48.954257 28346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:48.948926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:50333)
0924 20:20:48.949218 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:48.949231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:48.949326 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:48.949577 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:48.949580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:48.949756 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:48.949857 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.949862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.950557 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.950560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.951133 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:48.951140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.951531 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.951533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:48.951588 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.952076 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:48.952087 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:48.954086 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":109,"threads_started":1}
W20260924 20:20:48.954320 28344 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:48.949021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:50333 (local address 127.18.172.126:35253)
0924 20:20:48.949154 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:48.949158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:48.949282 (+ 124us) server_negotiation.cc:407] Connection header received
0924 20:20:48.949392 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:48.949394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:48.949437 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:48.949505 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:48.949965 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.950467 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.951229 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:48.951440 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:48.951516 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:48.954210 (+ 2694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:50333: BlockingRecv error: recv got EOF from 127.18.172.67:50333 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:49.554448 28347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:49.548564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:44643)
0924 20:20:49.548878 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:49.548893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:49.548992 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:49.549291 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:49.549293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:49.549479 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:49.549581 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.549586 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.550420 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.550423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.551135 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.551144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.551726 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.551728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.551842 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.552458 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:49.552470 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:49.554325 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":85,"threads_started":1}
W20260924 20:20:49.554677 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:49.548699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:44643 (local address 127.18.172.126:35253)
0924 20:20:49.548937 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:49.548941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:49.548957 (+ 16us) server_negotiation.cc:407] Connection header received
0924 20:20:49.549091 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:49.549093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:49.549140 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:49.549221 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:49.549691 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.550265 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.551242 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.551612 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.551717 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.554502 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:44643: BlockingRecv error: recv got EOF from 127.18.172.65:44643 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:49.743150 28349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:49.737852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:52075)
0924 20:20:49.738121 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:49.738134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:49.738237 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:49.738428 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:49.738431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:49.738609 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:49.738703 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.738708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.739535 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.739538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.740119 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.740128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.740568 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.740571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.740637 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.741160 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:49.741171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:49.743029 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":99,"threads_started":1}
W20260924 20:20:49.743299 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:49.737939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52075 (local address 127.18.172.126:35253)
0924 20:20:49.738078 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:49.738082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:49.738183 (+ 101us) server_negotiation.cc:407] Connection header received
0924 20:20:49.738307 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:49.738309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:49.738349 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:49.738415 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:49.738809 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.739409 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.740236 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.740464 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.740544 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.743184 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:52075: BlockingRecv error: recv got EOF from 127.18.172.66:52075 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:49 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:49.966022 28350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:49.955206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:52489)
0924 20:20:49.955621 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:49.955639 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:49.955804 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:49.956073 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:49.956075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:49.956347 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:49.956448 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.956455 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.957317 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.957324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.957890 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:49.957899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.958380 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.958383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:49.958452 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.964018 (+ 5566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:49.964032 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:49.965898 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":85,"threads_started":1}
W20260924 20:20:49.966161 28348 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:49.955339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:52489 (local address 127.18.172.126:35253)
0924 20:20:49.955601 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:49.955606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:49.955748 (+ 142us) server_negotiation.cc:407] Connection header received
0924 20:20:49.955915 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:49.955923 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:49.955971 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:49.956052 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:49.956586 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.957135 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.958019 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:49.958251 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:49.958344 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:49.966040 (+ 7696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:52489: BlockingRecv error: recv got EOF from 127.18.172.67:52489 (error 108)
Metrics: {"server-negotiator.queue_time_us":161}
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:50.561200 28351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:50.555466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:56353)
0924 20:20:50.555780 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:50.555794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:50.555894 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:50.556201 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:50.556204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:50.556442 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:50.556539 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.556544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.557366 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.557369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.557930 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.557938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.558520 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.558522 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.558586 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.559191 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:50.559203 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:50.561085 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":88,"threads_started":1}
W20260924 20:20:50.561506 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:50.555547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:56353 (local address 127.18.172.126:35253)
0924 20:20:50.555812 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:50.555815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:50.555848 (+ 33us) server_negotiation.cc:407] Connection header received
0924 20:20:50.555989 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:50.555991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:50.556038 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:50.556126 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:50.556665 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.557260 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.558060 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.558406 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.558492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.561367 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:56353: BlockingRecv error: recv got EOF from 127.18.172.65:56353 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:50.749485 28353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:50.743998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:45765)
0924 20:20:50.744300 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:50.744316 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:50.744447 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:50.744747 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:50.744749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:50.744958 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:50.745055 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.745061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.745887 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.745889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.746458 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.746466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.746879 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.746881 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.746940 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.747462 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:50.747474 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:50.749319 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":101,"threads_started":1}
W20260924 20:20:50.749558 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:50.744153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:45765 (local address 127.18.172.126:35253)
0924 20:20:50.744301 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:50.744307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:50.744373 (+ 66us) server_negotiation.cc:407] Connection header received
0924 20:20:50.744552 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:50.744555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:50.744607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:50.744702 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:50.745162 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.745772 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.746559 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.746774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.746849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.749448 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:45765: BlockingRecv error: recv got EOF from 127.18.172.66:45765 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:50 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:50.972589 28354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:50.966843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:52415)
0924 20:20:50.967178 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:50.967193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:50.967336 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:50.967595 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:50.967598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:50.967834 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:50.967924 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.967929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.968671 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.968676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.969274 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:50.969283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.969726 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.969730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:50.969812 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.970380 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:50.970392 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:50.972470 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
W20260924 20:20:50.972771 28352 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:50.966951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:52415 (local address 127.18.172.126:35253)
0924 20:20:50.967133 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:50.967138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:50.967289 (+ 151us) server_negotiation.cc:407] Connection header received
0924 20:20:50.967429 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:50.967433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:50.967487 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:50.967574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:50.968040 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.968564 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.969395 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:50.969617 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:50.969722 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:50.972604 (+ 2882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:52415: BlockingRecv error: recv got EOF from 127.18.172.67:52415 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:51.567654 28355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:51.562054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:55541)
0924 20:20:51.562346 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:51.562363 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:51.562487 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:51.562767 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:51.562769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:51.562956 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:51.563082 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.563088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.563856 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.563859 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.564500 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.564517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.564935 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.564938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.565000 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.565478 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:51.565490 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:51.567469 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":281,"threads_started":1}
W20260924 20:20:51.567729 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:51.562208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:55541 (local address 127.18.172.126:35253)
0924 20:20:51.562541 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:51.562546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:51.562559 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:20:51.562596 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:51.562598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:51.562641 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:51.562727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:51.563190 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.563744 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.564618 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.564835 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.564916 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.567615 (+ 2699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:55541: BlockingRecv error: recv got EOF from 127.18.172.65:55541 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":89,"threads_started":1}
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:51.756510 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:51.750390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:43027)
0924 20:20:51.750653 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:51.750666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:51.750765 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:51.751037 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:51.751043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:51.751229 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:51.751392 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.751398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.752125 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.752127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.752746 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.752762 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.753506 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.753512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.753613 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.754209 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:51.754224 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:51.756351 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":69,"threads_started":1}
W20260924 20:20:51.756664 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:51.750479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:43027 (local address 127.18.172.126:35253)
0924 20:20:51.750661 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:51.750665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:51.750743 (+ 78us) server_negotiation.cc:407] Connection header received
0924 20:20:51.750844 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:51.750847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:51.750892 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:51.750960 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:51.751506 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.752032 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.752906 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.753352 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.753462 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.756511 (+ 3049us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:43027: BlockingRecv error: recv got EOF from 127.18.172.66:43027 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:51 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.67@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:51.979938 28358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:51.973506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.67:43677)
0924 20:20:51.973824 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:51.973844 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:51.973969 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:51.974357 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:51.974360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:51.974588 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:51.974748 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.974758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.975688 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.975691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.976285 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:51.976295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.976838 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.976842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:51.976928 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.977793 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:51.977805 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:51.979821 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":141,"threads_started":1}
W20260924 20:20:51.980144 28356 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:51.973598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:43677 (local address 127.18.172.126:35253)
0924 20:20:51.973735 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:51.973740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:51.973928 (+ 188us) server_negotiation.cc:407] Connection header received
0924 20:20:51.974077 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:51.974080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:51.974139 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:51.974339 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:51.974892 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.975574 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.976413 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:51.976728 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:51.976835 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:51.979997 (+ 3162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.67:43677: BlockingRecv error: recv got EOF from 127.18.172.67:43677 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 20:20:52 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:52 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.65@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:52.574796 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:52.568749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.65:33009)
0924 20:20:52.569055 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:52.569072 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:52.569212 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:52.569558 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:52.569561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:52.569835 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:52.569969 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:52.569978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.570965 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.570967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:52.571599 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:52.571608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.572061 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.572064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:52.572128 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:52.572716 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:52.572731 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:52.574678 (+ 1947us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":129,"threads_started":1}
W20260924 20:20:52.575165 28360 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:52.568939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33009 (local address 127.18.172.126:35253)
0924 20:20:52.569215 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:52.569219 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:52.569233 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:20:52.569305 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:52.569307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:52.569354 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:52.569498 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:52.570092 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.570856 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.571724 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.571960 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.572044 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:52.574964 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.65:33009: BlockingRecv error: recv got EOF from 127.18.172.65:33009 (error 108)
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":81,"threads_started":1}
Sep 24 20:20:52 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 20:20:52 dist-test-slave-m1qq krb5kdc[27298](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.18.172.66@KRBTEST.COM for oryx/127.18.172.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 20:20:52.763628 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:52.757667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35253 (local address 127.18.172.66:57761)
0924 20:20:52.758004 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:52.758021 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:52.758174 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:52.758471 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:52.758474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:52.758703 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:52.758829 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:52.758836 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.759805 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.759808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:52.760547 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:52.760558 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.761147 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.761150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:52.761215 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:52.761671 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:52.761683 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:52.763507 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.18.172.126:35253: Server oryx/127.18.172.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
W20260924 20:20:52.764010 28360 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:52.757840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57761 (local address 127.18.172.126:35253)
0924 20:20:52.758006 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:52.758012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:52.758101 (+ 89us) server_negotiation.cc:407] Connection header received
0924 20:20:52.758295 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:52.758298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:52.758359 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:52.758448 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:52.758939 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.759668 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.760701 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:52.761056 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:52.761139 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:52.763712 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.18.172.66:57761: BlockingRecv error: recv got EOF from 127.18.172.66:57761 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:20:52.854523 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27881
I20260924 20:20:52.860078 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28034
I20260924 20:20:52.865417 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28167
I20260924 20:20:52.870414 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 27811
2026-09-24T20:20:52Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18088 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T20:20:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:52Z Disabled control of system clock
I20260924 20:20:52.902761 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43743
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 20:20:53.015972 28365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:53.016290 28365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:53.016328 28365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:53.021049 28365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:53.021116 28365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:53.021134 28365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:53.021159 28365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:53.021178 28365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:53.026502 28365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:53.027777 28365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:53.029194 28365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:53.038539 28371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.038569 28370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.038594 28373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.038686 28365 server_base.cc:1034] running on GCE node
I20260924 20:20:53.039465 28365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:53.040625 28365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:53.041831 28365 hybrid_clock.cc:648] HybridClock initialized: now 1790281253041797 us; error 52 us; skew 500 ppm
I20260924 20:20:53.043820 28365 webserver.cc:460] Webserver started at http://127.18.172.126:38803/ using document root <none> and password file <none>
I20260924 20:20:53.044417 28365 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:53.044513 28365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:53.044750 28365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:53.046919 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "7e9943a62c164a9a8c16b0fcdf7d537b"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "46a07e3787c1f5b90337522f0c3b7250"
server_key_iv: "a233de991c42bb526566a24b26eefcfb"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.047546 28365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "7e9943a62c164a9a8c16b0fcdf7d537b"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "46a07e3787c1f5b90337522f0c3b7250"
server_key_iv: "a233de991c42bb526566a24b26eefcfb"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.051290 28365 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:20:53.053634 28379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.054935 28365 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:53.055120 28365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "7e9943a62c164a9a8c16b0fcdf7d537b"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "46a07e3787c1f5b90337522f0c3b7250"
server_key_iv: "a233de991c42bb526566a24b26eefcfb"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.055238 28365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:53.067783 28365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:53.068514 28365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:53.068718 28365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:53.075927 28365 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43743
I20260924 20:20:53.075937 28431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43743 every 8 connection(s)
I20260924 20:20:53.076908 28365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:53.078750 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28365
I20260924 20:20:53.078884 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:53.079267 19121 external_mini_cluster.cc:1428] Setting key 6c8a541dadebdf93291d78052611587a
I20260924 20:20:53.080835 28432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:53.086416 28432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b: Bootstrap starting.
I20260924 20:20:53.088675 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.080772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.18.172.126:43743)
0924 20:20:53.081420 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.081436 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.081471 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:20:53.082132 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.082148 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.082550 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.082918 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.083390 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.084348 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.085238 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.085709 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.085889 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.086837 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:53.086842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:53.086854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:53.087350 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:53.087407 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:53.087428 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:53.087534 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:53.087697 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 20:20:53.087964 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260924 20:20:53.088822 28432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:53.089710 28432 log.cc:824] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:53.091905 28432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b: No bootstrap required, opened a new log
I20260924 20:20:53.095566 28432 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } }
I20260924 20:20:53.095850 28432 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:53.095913 28432 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e9943a62c164a9a8c16b0fcdf7d537b, State: Initialized, Role: FOLLOWER
I20260924 20:20:53.096472 28432 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } }
I20260924 20:20:53.096691 28432 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:53.096804 28432 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:53.096935 28432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:53.097975 28432 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } }
I20260924 20:20:53.098342 28432 leader_election.cc:302] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e9943a62c164a9a8c16b0fcdf7d537b; no voters:
I20260924 20:20:53.098675 28432 leader_election.cc:288] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:53.098803 28437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:53.099094 28437 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [term 1 LEADER]: Becoming Leader. State: Replica: 7e9943a62c164a9a8c16b0fcdf7d537b, State: Running, Role: LEADER
I20260924 20:20:53.099462 28437 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } }
I20260924 20:20:53.100046 28432 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:53.101354 28439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e9943a62c164a9a8c16b0fcdf7d537b. Latest consensus state: current_term: 1 leader_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } } }
I20260924 20:20:53.101503 28439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:53.101354 28438 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e9943a62c164a9a8c16b0fcdf7d537b" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43743 } } }
I20260924 20:20:53.102336 28438 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:53.102707 28452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:53.105657 28452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:53.110710 28452 catalog_manager.cc:1347] Generated new cluster ID: 16188f5482c145deb0f2c46194bc2f57
I20260924 20:20:53.110776 28452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:53.133574 28452 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:53.134905 28452 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:53.145011 28452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e9943a62c164a9a8c16b0fcdf7d537b: Generated new TSK 0
I20260924 20:20:53.145999 28452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:53.161110 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--builtin_ntp_servers=127.18.172.84:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 20:20:53.274250 28456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:53.274539 28456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:53.274576 28456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:53.278740 28456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:53.278802 28456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:53.278856 28456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:53.283875 28456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:53.285032 28456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:53.286494 28456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:53.297919 28464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.297932 28462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.297932 28461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.298224 28456 server_base.cc:1034] running on GCE node
I20260924 20:20:53.298671 28456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:53.299379 28456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:53.300534 28456 hybrid_clock.cc:648] HybridClock initialized: now 1790281253300514 us; error 32 us; skew 500 ppm
I20260924 20:20:53.302469 28456 webserver.cc:460] Webserver started at http://127.18.172.65:39611/ using document root <none> and password file <none>
I20260924 20:20:53.303072 28456 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:53.303139 28456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:53.303310 28456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:53.305389 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "0c9d9110b7ef449384119a5af2f1eda7"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "bbe872d613e5183718f8593c932db6ae"
server_key_iv: "c8fdf1845047948828f76560536be138"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.305893 28456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "0c9d9110b7ef449384119a5af2f1eda7"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "bbe872d613e5183718f8593c932db6ae"
server_key_iv: "c8fdf1845047948828f76560536be138"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.309566 28456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:53.312530 28471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.313892 28456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:20:53.313987 28456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "0c9d9110b7ef449384119a5af2f1eda7"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "bbe872d613e5183718f8593c932db6ae"
server_key_iv: "c8fdf1845047948828f76560536be138"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.314083 28456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:53.328998 28456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:53.329630 28456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:53.329790 28456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:53.330315 28456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:53.331259 28456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:53.331327 28456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.331369 28456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:53.331390 28456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.340476 28456 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:45115
I20260924 20:20:53.340556 28584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:45115 every 8 connection(s)
I20260924 20:20:53.341434 28456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:53.347822 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28456
I20260924 20:20:53.347951 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:53.348265 19121 external_mini_cluster.cc:1428] Setting key 91c258fc39cf321d32d27316b9079c84
I20260924 20:20:53.350483 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--builtin_ntp_servers=127.18.172.84:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:20:53.351974 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.343390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:37685 (local address 127.18.172.126:43743)
0924 20:20:53.343530 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.343534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.344759 (+ 1225us) server_negotiation.cc:407] Connection header received
0924 20:20:53.346255 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.346259 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.346325 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.346439 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.347470 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.348311 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.349629 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.350062 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.350185 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.351402 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:53.351408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:53.351414 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:53.351432 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:53.351460 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:53.351474 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:53.351566 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:53.351736 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 20:20:53.351828 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:20:53.352840 28587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.343827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43743 (local address 127.18.172.65:37685)
0924 20:20:53.344552 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:53.344597 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:53.346016 (+ 1419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:53.346636 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:53.346644 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:53.346733 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:53.347316 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.347331 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.348456 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.348459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.349477 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.349487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.350214 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.350217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.350399 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.351129 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:53.351131 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:53.351174 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:53.351241 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:53.351572 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:53.351579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:53.351583 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:20:53.351715 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 20:20:53.352004 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":103,"threads_started":1}
I20260924 20:20:53.354360 28585 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43743
I20260924 20:20:53.354645 28585 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:53.355333 28585 heartbeater.cc:511] Master 127.18.172.126:43743 requested a full tablet report, sending...
I20260924 20:20:53.357157 28396 ts_manager.cc:194] Registered new tserver with Master: 0c9d9110b7ef449384119a5af2f1eda7 (127.18.172.65:45115)
I20260924 20:20:53.358599 28396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.65:37685
W20260924 20:20:53.464491 28588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:53.464823 28588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:53.464898 28588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:53.469064 28588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:53.469313 28588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:53.469421 28588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:53.474263 28588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:53.475435 28588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:53.476873 28588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:53.487030 28594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.487063 28593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.487125 28588 server_base.cc:1034] running on GCE node
W20260924 20:20:53.487063 28596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.487730 28588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:53.488314 28588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:53.489476 28588 hybrid_clock.cc:648] HybridClock initialized: now 1790281253489459 us; error 28 us; skew 500 ppm
I20260924 20:20:53.491464 28588 webserver.cc:460] Webserver started at http://127.18.172.66:41357/ using document root <none> and password file <none>
I20260924 20:20:53.492030 28588 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:53.492089 28588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:53.492440 28588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:53.494598 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "45994f101bce4f90923dbf73349f8dbd"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "5f161b2d749c316e72ed44a53485189a"
server_key_iv: "af8b0b88d91f2929af488b3468071417"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.495234 28588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "45994f101bce4f90923dbf73349f8dbd"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "5f161b2d749c316e72ed44a53485189a"
server_key_iv: "af8b0b88d91f2929af488b3468071417"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.498924 28588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:53.501976 28602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.503495 28588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:20:53.503634 28588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "45994f101bce4f90923dbf73349f8dbd"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "5f161b2d749c316e72ed44a53485189a"
server_key_iv: "af8b0b88d91f2929af488b3468071417"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.503758 28588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:53.517422 28588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:53.518187 28588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:53.518391 28588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:53.518972 28588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:53.519948 28588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:53.520020 28588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.520093 28588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:53.520140 28588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.528709 28588 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:46355
I20260924 20:20:53.528746 28716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:46355 every 8 connection(s)
I20260924 20:20:53.529670 28588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:53.537889 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28588
I20260924 20:20:53.538028 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:53.538471 19121 external_mini_cluster.cc:1428] Setting key 753c31075eb61b4458c76e8f1eaf32b0
I20260924 20:20:53.539578 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.531539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:38207 (local address 127.18.172.126:43743)
0924 20:20:53.531680 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.531685 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.532858 (+ 1173us) server_negotiation.cc:407] Connection header received
0924 20:20:53.534347 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.534350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.534398 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.534475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.535528 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.536074 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.537326 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.537573 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.537661 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.539049 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:53.539055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:53.539059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:53.539074 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:53.539095 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:53.539105 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:53.539181 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:53.539363 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 20:20:53.539445 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:53.540660 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.531995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43743 (local address 127.18.172.66:38207)
0924 20:20:53.532708 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:53.532751 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:53.534125 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:53.534666 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:53.534673 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:53.534761 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:53.535393 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.535410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.536211 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.536215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.537196 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.537205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.537766 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.537771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.537992 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.538704 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:53.538707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:53.538765 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:53.538846 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:53.539202 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:53.539211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:53.539217 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 20:20:53.539355 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:20:53.539611 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":113,"threads_started":1}
I20260924 20:20:53.540817 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--builtin_ntp_servers=127.18.172.84:32995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 20:20:53.541877 28717 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43743
I20260924 20:20:53.542135 28717 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:53.542721 28717 heartbeater.cc:511] Master 127.18.172.126:43743 requested a full tablet report, sending...
I20260924 20:20:53.543968 28396 ts_manager.cc:194] Registered new tserver with Master: 45994f101bce4f90923dbf73349f8dbd (127.18.172.66:46355)
I20260924 20:20:53.544816 28396 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.66:38207
W20260924 20:20:53.652314 28720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:53.652576 28720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:53.652613 28720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:53.656548 28720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:53.656601 28720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:53.656680 28720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:53.661497 28720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:53.662539 28720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:53.663998 28720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:53.673980 28728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.673970 28725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.674032 28720 server_base.cc:1034] running on GCE node
W20260924 20:20:53.673974 28726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.674799 28720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:53.675503 28720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:53.676702 28720 hybrid_clock.cc:648] HybridClock initialized: now 1790281253676684 us; error 28 us; skew 500 ppm
I20260924 20:20:53.678656 28720 webserver.cc:460] Webserver started at http://127.18.172.67:45873/ using document root <none> and password file <none>
I20260924 20:20:53.679322 28720 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:53.679412 28720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:53.679649 28720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:53.681758 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "10744d2d465c46aa813c74f1a1a9f6e3"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "27aa06f343046bbc2bf428ddee8f3999"
server_key_iv: "72abf805e39b5bcfeb7fd7154d29b7ae"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.682323 28720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "10744d2d465c46aa813c74f1a1a9f6e3"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "27aa06f343046bbc2bf428ddee8f3999"
server_key_iv: "72abf805e39b5bcfeb7fd7154d29b7ae"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.686086 28720 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:53.688827 28734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.690254 28720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:53.690397 28720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "10744d2d465c46aa813c74f1a1a9f6e3"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "27aa06f343046bbc2bf428ddee8f3999"
server_key_iv: "72abf805e39b5bcfeb7fd7154d29b7ae"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.690518 28720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:53.705257 28720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:53.705960 28720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:53.706154 28720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:53.706777 28720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:53.707931 28720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:53.708000 28720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.708066 28720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:53.708113 28720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.716948 28720 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:35793
I20260924 20:20:53.716975 28849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:35793 every 8 connection(s)
I20260924 20:20:53.718005 28720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:53.727164 28435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.719976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:59897 (local address 127.18.172.126:43743)
0924 20:20:53.720112 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.720116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.721163 (+ 1047us) server_negotiation.cc:407] Connection header received
0924 20:20:53.722528 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.722531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.722576 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.722650 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.723633 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.724149 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.725281 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.725503 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.725583 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.726630 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:53.726633 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:53.726635 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:53.726646 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:53.726662 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:53.726669 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:53.726727 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:53.726931 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 20:20:53.726979 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:53.727887 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28720
I20260924 20:20:53.728027 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:53.728299 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.720379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43743 (local address 127.18.172.67:59897)
0924 20:20:53.721013 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:53.721055 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:53.722335 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:53.722818 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:53.722825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:53.722906 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:53.723513 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.723526 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.724301 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.724304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.725180 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:53.725188 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:53.725633 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:53.725636 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:53.725772 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:53.726396 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:53.726398 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:53.726437 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:53.726496 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:53.726772 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:53.726780 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:53.726785 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:20:53.726921 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 20:20:53.727254 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":106,"threads_started":1}
I20260924 20:20:53.728335 19121 external_mini_cluster.cc:1428] Setting key 0d802cd9692e419601de02f7c4a513b3
I20260924 20:20:53.729532 28850 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43743
I20260924 20:20:53.729790 28850 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:53.730408 28850 heartbeater.cc:511] Master 127.18.172.126:43743 requested a full tablet report, sending...
I20260924 20:20:53.731473 28395 ts_manager.cc:194] Registered new tserver with Master: 10744d2d465c46aa813c74f1a1a9f6e3 (127.18.172.67:35793)
I20260924 20:20:53.732028 28395 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.67:59897
I20260924 20:20:53.742686 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:20:53.745899 28435 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:53.744727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59448 (local address 127.18.172.126:43743)
0924 20:20:53.744864 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.744869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.745079 (+ 210us) server_negotiation.cc:407] Connection header received
0924 20:20:53.745333 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.745336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.745386 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.745469 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.745761 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59448: BlockingRecv error: recv got EOF from 127.0.0.1:59448 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:20:53.746047 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28456
I20260924 20:20:53.751768 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28588
I20260924 20:20:53.757023 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28720
I20260924 20:20:53.762892 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28365
2026-09-24T20:20:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (890 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T20:20:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:53Z Disabled control of system clock
I20260924 20:20:53.793092 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36859
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:36859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:20:53.903898 28861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:53.904181 28861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:53.904227 28861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:53.908293 28861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:53.908350 28861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:53.908366 28861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:53.908384 28861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:53.908399 28861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:53.913225 28861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:36859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:36859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:53.914284 28861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:53.915752 28861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:53.924705 28867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:53.924686 28866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.924976 28861 server_base.cc:1034] running on GCE node
W20260924 20:20:53.924667 28869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:53.925721 28861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:53.926888 28861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:53.928083 28861 hybrid_clock.cc:648] HybridClock initialized: now 1790281253928065 us; error 41 us; skew 500 ppm
I20260924 20:20:53.930025 28861 webserver.cc:460] Webserver started at http://127.18.172.126:42089/ using document root <none> and password file <none>
I20260924 20:20:53.930629 28861 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:53.930721 28861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:53.930946 28861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:53.933115 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "44b7318743084f8c90208618130512f4"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "1919d0a66b159c2417a74c61457a432c"
server_key_iv: "c8cf3e967cf6141052b5d3dfe6e073ad"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.933717 28861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "44b7318743084f8c90208618130512f4"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "1919d0a66b159c2417a74c61457a432c"
server_key_iv: "c8cf3e967cf6141052b5d3dfe6e073ad"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.937678 28861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:20:53.940243 28875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:53.941509 28861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:20:53.941653 28861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "44b7318743084f8c90208618130512f4"
format_stamp: "Formatted at 2026-09-24 20:20:53 on dist-test-slave-m1qq"
server_key: "1919d0a66b159c2417a74c61457a432c"
server_key_iv: "c8cf3e967cf6141052b5d3dfe6e073ad"
server_key_version: "encryptionkey@0"
I20260924 20:20:53.941776 28861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:53.946466 28861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:53.947144 28861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:53.947332 28861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:53.954181 28861 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:36859
I20260924 20:20:53.954156 28928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:36859 every 8 connection(s)
I20260924 20:20:53.955171 28861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:53.958321 28929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:53.958995 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28861
I20260924 20:20:53.959133 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:53.959460 19121 external_mini_cluster.cc:1428] Setting key 3333fa8c413fb60e3d8d664b6f506906
I20260924 20:20:53.964728 28929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4: Bootstrap starting.
I20260924 20:20:53.965085 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:53.960879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35618 (local address 127.18.172.126:36859)
0924 20:20:53.961501 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:53.961517 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:53.961551 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:20:53.962213 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:53.962231 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:53.962571 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:53.962864 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:53.963731 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:53.963735 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:53.963747 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:53.963757 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:53.963814 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:53.963829 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:53.963938 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:53.964063 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 20:20:53.964337 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":137,"threads_started":1}
I20260924 20:20:53.966917 28929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:53.967679 28929 log.cc:824] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:53.969566 28929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4: No bootstrap required, opened a new log
I20260924 20:20:53.972987 28929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } }
I20260924 20:20:53.973234 28929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:53.973320 28929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44b7318743084f8c90208618130512f4, State: Initialized, Role: FOLLOWER
I20260924 20:20:53.973730 28929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } }
I20260924 20:20:53.973883 28929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:53.973968 28929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:53.974085 28929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:53.974962 28929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } }
I20260924 20:20:53.975322 28929 leader_election.cc:302] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44b7318743084f8c90208618130512f4; no voters:
I20260924 20:20:53.975622 28929 leader_election.cc:288] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:53.975751 28934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:53.975999 28934 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [term 1 LEADER]: Becoming Leader. State: Replica: 44b7318743084f8c90208618130512f4, State: Running, Role: LEADER
I20260924 20:20:53.976308 28934 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } }
I20260924 20:20:53.976918 28929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:53.977908 28936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44b7318743084f8c90208618130512f4. Latest consensus state: current_term: 1 leader_uuid: "44b7318743084f8c90208618130512f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } } }
I20260924 20:20:53.978042 28936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:53.978379 28943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:53.978387 28935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44b7318743084f8c90208618130512f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44b7318743084f8c90208618130512f4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 36859 } } }
I20260924 20:20:53.978504 28935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:53.981284 28943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:53.986418 28943 catalog_manager.cc:1347] Generated new cluster ID: 24ca269c6ac942969336247bc53f1750
I20260924 20:20:53.986521 28943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:53.994454 28943 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:53.994767 28943 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:54.005203 28943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44b7318743084f8c90208618130512f4: Generated new TSK 0
I20260924 20:20:54.006086 28943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:20:54.016474 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--builtin_ntp_servers=127.18.172.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:20:54.146340 28953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:54.146632 28953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:54.146705 28953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:54.150840 28953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:54.150934 28953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:54.151078 28953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:54.156248 28953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:54.157450 28953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:54.158898 28953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:54.168972 28958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:54.168994 28959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:54.168994 28961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:54.169263 28953 server_base.cc:1034] running on GCE node
I20260924 20:20:54.169834 28953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:54.170517 28953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:54.171692 28953 hybrid_clock.cc:648] HybridClock initialized: now 1790281254171661 us; error 43 us; skew 500 ppm
I20260924 20:20:54.174016 28953 webserver.cc:460] Webserver started at http://127.18.172.65:40173/ using document root <none> and password file <none>
I20260924 20:20:54.174669 28953 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:54.174736 28953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:54.174973 28953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:54.177330 28953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "57bf80e388ae439c9eef52c70b242431"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "ae8b60cb75047da0fc6d20da61261467"
server_key_iv: "441d70de859576d1ed9657aba5c0efdc"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.177937 28953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "57bf80e388ae439c9eef52c70b242431"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "ae8b60cb75047da0fc6d20da61261467"
server_key_iv: "441d70de859576d1ed9657aba5c0efdc"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.182118 28953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:20:54.185186 28967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.186628 28953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:54.186769 28953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "57bf80e388ae439c9eef52c70b242431"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "ae8b60cb75047da0fc6d20da61261467"
server_key_iv: "441d70de859576d1ed9657aba5c0efdc"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.186894 28953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:54.191999 28953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:54.192600 28953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:54.192777 28953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:54.193316 28953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:54.194236 28953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:54.194289 28953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.194329 28953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:54.194386 28953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.207248 28953 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:43227
I20260924 20:20:54.207297 29081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:43227 every 8 connection(s)
I20260924 20:20:54.208235 28953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:54.214309 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 28953
I20260924 20:20:54.214439 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:54.214771 19121 external_mini_cluster.cc:1428] Setting key 84a14ae15f2e578ad6470af04b0c3e4d
I20260924 20:20:54.216451 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.210514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:57207 (local address 127.18.172.126:36859)
0924 20:20:54.213108 (+ 2594us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:54.213113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:54.213131 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:20:54.214026 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:54.214029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:54.214073 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:54.214156 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:54.215867 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:54.215871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:54.215875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:54.215919 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:54.215947 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:54.215961 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:54.216065 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:54.216277 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 20:20:54.216326 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2492}
I20260924 20:20:54.217023 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--builtin_ntp_servers=127.18.172.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 20:20:54.217808 29084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.210967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36859 (local address 127.18.172.65:57207)
0924 20:20:54.211887 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:54.211948 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:54.213813 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:54.214455 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:54.214465 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:54.214584 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:54.215503 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:54.215505 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:54.215557 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:54.215646 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:54.216107 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:54.216120 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:54.216130 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 20:20:54.216321 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 20:20:54.216579 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":107,"threads_started":1}
I20260924 20:20:54.219234 29082 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36859
I20260924 20:20:54.219496 29082 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:54.220098 29082 heartbeater.cc:511] Master 127.18.172.126:36859 requested a full tablet report, sending...
I20260924 20:20:54.221848 28892 ts_manager.cc:194] Registered new tserver with Master: 57bf80e388ae439c9eef52c70b242431 (127.18.172.65:43227)
W20260924 20:20:54.339939 29085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:54.340214 29085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:54.340255 29085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:54.344393 29085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:54.344457 29085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:54.344534 29085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:54.349509 29085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:54.350610 29085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:54.352348 29085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:54.361961 29093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:54.361961 29090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:54.362118 29085 server_base.cc:1034] running on GCE node
W20260924 20:20:54.361924 29091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:54.362754 29085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:54.363512 29085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:54.364717 29085 hybrid_clock.cc:648] HybridClock initialized: now 1790281254364671 us; error 61 us; skew 500 ppm
I20260924 20:20:54.366911 29085 webserver.cc:460] Webserver started at http://127.18.172.66:41999/ using document root <none> and password file <none>
I20260924 20:20:54.367540 29085 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:54.367605 29085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:54.367782 29085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:54.369838 29085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "94e41a206c1840ec8097048411f53789"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "e760476c95ed1f57f0a800ee00d96225"
server_key_iv: "79fb44bb27bec7dca839297977c99dba"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.370405 29085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "94e41a206c1840ec8097048411f53789"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "e760476c95ed1f57f0a800ee00d96225"
server_key_iv: "79fb44bb27bec7dca839297977c99dba"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.374183 29085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260924 20:20:54.376983 29099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.378415 29085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:20:54.378509 29085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "94e41a206c1840ec8097048411f53789"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "e760476c95ed1f57f0a800ee00d96225"
server_key_iv: "79fb44bb27bec7dca839297977c99dba"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.378646 29085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:54.383390 29085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:54.384014 29085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:54.384212 29085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:54.384768 29085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:54.385712 29085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:54.385782 29085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.385854 29085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:54.385907 29085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.395424 29085 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:46139
I20260924 20:20:54.395443 29213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:46139 every 8 connection(s)
I20260924 20:20:54.396463 29085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:54.402990 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.398382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:45041 (local address 127.18.172.126:36859)
0924 20:20:54.398536 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:54.398540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:54.399544 (+ 1004us) server_negotiation.cc:407] Connection header received
0924 20:20:54.401052 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:54.401055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:54.401102 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:54.401309 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:54.402481 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:54.402484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:54.402487 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:54.402499 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:54.402521 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:54.402530 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:54.402635 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:54.402810 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 20:20:54.402858 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 20:20:54.403843 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.398708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36859 (local address 127.18.172.66:45041)
0924 20:20:54.399403 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:54.399442 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:54.400694 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:54.401486 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:54.401493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:54.401581 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:54.402218 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:54.402220 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:54.402262 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:54.402328 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:54.402654 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:54.402665 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:54.402676 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 20:20:54.402840 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 20:20:54.403104 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260924 20:20:54.404956 29214 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36859
I20260924 20:20:54.405166 29214 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:54.405547 29214 heartbeater.cc:511] Master 127.18.172.126:36859 requested a full tablet report, sending...
I20260924 20:20:54.406525 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29085
I20260924 20:20:54.406553 28892 ts_manager.cc:194] Registered new tserver with Master: 94e41a206c1840ec8097048411f53789 (127.18.172.66:46139)
I20260924 20:20:54.406662 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:54.406971 19121 external_mini_cluster.cc:1428] Setting key cd4a6d46bfc7357dda822ac42af3480f
I20260924 20:20:54.409097 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--builtin_ntp_servers=127.18.172.84:44725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 20:20:54.519225 29217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:54.519550 29217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:54.519620 29217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:54.523723 29217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:54.523814 29217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:54.523901 29217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:54.528753 29217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:36859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29217
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:54.529834 29217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:54.531323 29217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:54.541178 29222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:54.541217 29223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:54.541277 29217 server_base.cc:1034] running on GCE node
W20260924 20:20:54.541182 29225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:54.541941 29217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:54.542537 29217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:54.543743 29217 hybrid_clock.cc:648] HybridClock initialized: now 1790281254543727 us; error 43 us; skew 500 ppm
I20260924 20:20:54.545833 29217 webserver.cc:460] Webserver started at http://127.18.172.67:35779/ using document root <none> and password file <none>
I20260924 20:20:54.546465 29217 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:54.546530 29217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:54.546705 29217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:54.548796 29217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "58b194651d2a4bbca9feabf8b6037de7"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "2b46643abda4e2b30dbe2a0f30e52b8d"
server_key_iv: "3e0db70f33d07b0ea84707e6bc575b06"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.549309 29217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "58b194651d2a4bbca9feabf8b6037de7"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "2b46643abda4e2b30dbe2a0f30e52b8d"
server_key_iv: "3e0db70f33d07b0ea84707e6bc575b06"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.553025 29217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 20:20:54.555646 29231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.556905 29217 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:20:54.557140 29217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "58b194651d2a4bbca9feabf8b6037de7"
format_stamp: "Formatted at 2026-09-24 20:20:54 on dist-test-slave-m1qq"
server_key: "2b46643abda4e2b30dbe2a0f30e52b8d"
server_key_iv: "3e0db70f33d07b0ea84707e6bc575b06"
server_key_version: "encryptionkey@0"
I20260924 20:20:54.557255 29217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:54.561825 29217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:54.562419 29217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:54.562599 29217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:54.563169 29217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:54.564054 29217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:54.564098 29217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.564165 29217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:54.564206 29217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:54.573087 29217 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:41227
I20260924 20:20:54.573104 29344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:41227 every 8 connection(s)
I20260924 20:20:54.574018 29217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:54.575177 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29217
I20260924 20:20:54.575294 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:54.575552 19121 external_mini_cluster.cc:1428] Setting key 016c4e10978ec899279400251acf01a7
I20260924 20:20:54.580570 28932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.576029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:58941 (local address 127.18.172.126:36859)
0924 20:20:54.576154 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:54.576166 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:54.577566 (+ 1400us) server_negotiation.cc:407] Connection header received
0924 20:20:54.578949 (+ 1383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:54.578952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:54.578988 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:54.579102 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:54.580156 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:54.580159 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:54.580161 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:20:54.580172 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:54.580190 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:20:54.580198 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:54.580269 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:54.580433 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 20:20:54.580470 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:20:54.581426 29347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:54.576575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:36859 (local address 127.18.172.67:58941)
0924 20:20:54.577406 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:54.577462 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:54.578769 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:54.579244 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:54.579251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:54.579365 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:54.579875 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:20:54.579876 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:54.579924 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:20:54.580025 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:54.580313 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:54.580319 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:54.580322 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 20:20:54.580450 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 20:20:54.580631 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":134,"threads_started":1}
I20260924 20:20:54.582458 29345 heartbeater.cc:347] Connected to a master server at 127.18.172.126:36859
I20260924 20:20:54.582669 29345 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:54.583194 29345 heartbeater.cc:511] Master 127.18.172.126:36859 requested a full tablet report, sending...
I20260924 20:20:54.584182 28892 ts_manager.cc:194] Registered new tserver with Master: 58b194651d2a4bbca9feabf8b6037de7 (127.18.172.67:41227)
I20260924 20:20:54.589619 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 20:20:54.592835 28932 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:20:54.591814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.18.172.126:36859)
0924 20:20:54.591994 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:54.592001 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:54.592182 (+ 181us) server_negotiation.cc:407] Connection header received
0924 20:20:54.592278 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:54.592281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:54.592316 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:54.592409 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:54.592710 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35624: BlockingRecv error: recv got EOF from 127.0.0.1:35624 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 20:20:54.592927 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28953
I20260924 20:20:54.598753 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29085
I20260924 20:20:54.603953 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29217
I20260924 20:20:54.609627 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 28861
2026-09-24T20:20:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (845 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:54 dist-test-slave-m1qq krb5kdc[29356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:54 dist-test-slave-m1qq krb5kdc[29356](info): set up 2 sockets
Sep 24 20:20:54 dist-test-slave-m1qq krb5kdc[29356](info): commencing operation
krb5kdc: starting...
W20260924 20:20:55.681363 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:55 dist-test-slave-m1qq krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:20:55.807317 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:42823
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:42823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:55.916687 29369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:55.917047 29369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:55.917164 29369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:55.921232 29369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:20:55.921324 29369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:55.921368 29369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:55.921411 29369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:20:55.921450 29369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:20:55.927062 29369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:42823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:42823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:55.928066 29369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:55.929061 29369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:55.937677 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:55.937855 29369 server_base.cc:1034] running on GCE node
W20260924 20:20:55.937678 29374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:55.937721 29375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:55.939596 29369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:55.940810 29369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:55.942027 29369 hybrid_clock.cc:648] HybridClock initialized: now 1790281255942012 us; error 31 us; skew 500 ppm
Sep 24 20:20:55 dist-test-slave-m1qq krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281255, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:55.950341 29369 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:20:55.951416 29369 webserver.cc:460] Webserver started at http://127.18.172.126:45219/ using document root <none> and password file <none>
I20260924 20:20:55.951962 29369 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:55.952020 29369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:55.952183 29369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:55.954228 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8"
format_stamp: "Formatted at 2026-09-24 20:20:55 on dist-test-slave-m1qq"
server_key: "f62ad2bb5330d2f5611e3ad2d4ecbd0e"
server_key_iv: "b35b161cd7583325e03db7a63a15cf79"
server_key_version: "encryptionkey@0"
I20260924 20:20:55.954737 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8"
format_stamp: "Formatted at 2026-09-24 20:20:55 on dist-test-slave-m1qq"
server_key: "f62ad2bb5330d2f5611e3ad2d4ecbd0e"
server_key_iv: "b35b161cd7583325e03db7a63a15cf79"
server_key_version: "encryptionkey@0"
I20260924 20:20:55.958386 29369 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:20:55.960758 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:55.962046 29369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:20:55.962153 29369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8"
format_stamp: "Formatted at 2026-09-24 20:20:55 on dist-test-slave-m1qq"
server_key: "f62ad2bb5330d2f5611e3ad2d4ecbd0e"
server_key_iv: "b35b161cd7583325e03db7a63a15cf79"
server_key_version: "encryptionkey@0"
I20260924 20:20:55.962231 29369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:55.977240 29369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:55.980448 29369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:55.980659 29369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:55.987671 29369 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:42823
I20260924 20:20:55.988001 29436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:42823 every 8 connection(s)
I20260924 20:20:55.988662 29369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:20:55.991540 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:55.993880 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29369
I20260924 20:20:55.994012 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:20:55.994313 19121 external_mini_cluster.cc:1428] Setting key dc00f891791af8df4b3410f8fec69724
I20260924 20:20:55.997810 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: Bootstrap starting.
I20260924 20:20:56.000386 29437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:56.001170 29437 log.cc:824] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:56.003646 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: No bootstrap required, opened a new log
I20260924 20:20:56.007495 29437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } }
I20260924 20:20:56.007812 29437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:56.007901 29437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ff149b4e83f4e6ba6a076593ea7f2c8, State: Initialized, Role: FOLLOWER
I20260924 20:20:56.008394 29437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } }
I20260924 20:20:56.008561 29437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:20:56.008653 29437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:20:56.008776 29437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:56.009815 29437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } }
I20260924 20:20:56.009929 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:55.995780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.18.172.126:42823)
0924 20:20:55.996192 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:55.996207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:55.996241 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:20:55.996872 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:55.996888 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:55.997222 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:55.997493 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:55.998350 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:55.999387 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.000560 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.001038 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.001163 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.004089 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:56.004094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:56.004107 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:56.004133 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:56.006658 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.007267 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.007282 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.007287 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.007404 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.007706 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.007709 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.007711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.008293 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:56.008513 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.008793 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.009139 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":130,"threads_started":1}
I20260924 20:20:56.010281 29437 leader_election.cc:302] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ff149b4e83f4e6ba6a076593ea7f2c8; no voters:
I20260924 20:20:56.010680 29437 leader_election.cc:288] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:20:56.010789 29442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:56.011032 29442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [term 1 LEADER]: Becoming Leader. State: Replica: 5ff149b4e83f4e6ba6a076593ea7f2c8, State: Running, Role: LEADER
I20260924 20:20:56.011356 29442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } }
I20260924 20:20:56.012096 29437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:20:56.012653 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ff149b4e83f4e6ba6a076593ea7f2c8. Latest consensus state: current_term: 1 leader_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } } }
I20260924 20:20:56.012850 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:56.013267 29447 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:20:56.013341 29443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ff149b4e83f4e6ba6a076593ea7f2c8" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 42823 } } }
I20260924 20:20:56.013543 29443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:20:56.016471 29447 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:20:56.022576 29447 catalog_manager.cc:1347] Generated new cluster ID: c72fbcfcc9b94fe7b496ff806d430ad3
I20260924 20:20:56.022665 29447 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:20:56.033442 29447 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:20:56.034426 29447 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:20:56.041117 29447 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: Generated new TSK 0
I20260924 20:20:56.041854 29447 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:20:56.074378 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:56.186631 29463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:56.186930 29463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:56.186975 29463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:56.191155 29463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:56.191221 29463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:56.191325 29463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:20:56.196805 29463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:56.197904 29463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:56.198956 29463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:56.208935 29468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:56.208935 29471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:56.209112 29469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:56.209147 29463 server_base.cc:1034] running on GCE node
I20260924 20:20:56.209830 29463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:56.210500 29463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:56.211705 29463 hybrid_clock.cc:648] HybridClock initialized: now 1790281256211693 us; error 25 us; skew 500 ppm
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:56.214725 29463 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:20:56.215662 29463 webserver.cc:460] Webserver started at http://127.18.172.65:44895/ using document root <none> and password file <none>
I20260924 20:20:56.216225 29463 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:56.216318 29463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:56.216552 29463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:56.218621 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "2c4115ffb99140768f82ef4c34c3ddf1"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "da088f7d0266b44478982f135168847c"
server_key_iv: "81d6f3d12dcaccac47bb91d0ff0e1ef9"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.219236 29463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "2c4115ffb99140768f82ef4c34c3ddf1"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "da088f7d0266b44478982f135168847c"
server_key_iv: "81d6f3d12dcaccac47bb91d0ff0e1ef9"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.223146 29463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:20:56.225543 29478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.226702 29463 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:56.226845 29463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "2c4115ffb99140768f82ef4c34c3ddf1"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "da088f7d0266b44478982f135168847c"
server_key_iv: "81d6f3d12dcaccac47bb91d0ff0e1ef9"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.226959 29463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:56.260815 29463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:56.264361 29463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:56.264581 29463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:56.265259 29463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:56.266183 29463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:56.266254 29463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.266320 29463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:56.266377 29463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.275825 29463 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:40419
I20260924 20:20:56.275846 29591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:40419 every 8 connection(s)
I20260924 20:20:56.276757 29463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:20:56.281951 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29463
I20260924 20:20:56.282091 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:20:56.282408 19121 external_mini_cluster.cc:1428] Setting key f022a557284c9e6e52b205397b42ae56
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:20:56.292366 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.278625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:48993 (local address 127.18.172.126:42823)
0924 20:20:56.278769 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.278773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.279758 (+ 985us) server_negotiation.cc:407] Connection header received
0924 20:20:56.281139 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.281142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.281191 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.281269 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:56.283257 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.283886 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.285153 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.285569 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.285696 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.288507 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:56.288511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:56.288513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:56.288536 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:56.290118 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.290749 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.290753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.290755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.290834 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.291238 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.291241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.291243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.291436 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:56.291539 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.292054 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.292181 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:20:56.293171 29594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.279052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:42823 (local address 127.18.172.65:48993)
0924 20:20:56.279564 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.279608 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.280911 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.281494 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.281501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.282324 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:56.282994 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.283065 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.284032 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.284037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.285004 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.285014 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.285801 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.285808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.286045 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.286705 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:56.286734 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:56.288324 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:56.290354 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.290360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.290378 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.290635 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.290924 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.290928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.290930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.291133 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.291548 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:56.291556 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:56.291815 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.292002 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.292337 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":116,"threads_started":1}
I20260924 20:20:56.294438 29592 heartbeater.cc:347] Connected to a master server at 127.18.172.126:42823
I20260924 20:20:56.294713 29592 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:20:56.295326 29592 heartbeater.cc:511] Master 127.18.172.126:42823 requested a full tablet report, sending...
I20260924 20:20:56.296927 29401 ts_manager.cc:194] Registered new tserver with Master: 2c4115ffb99140768f82ef4c34c3ddf1 (127.18.172.65:40419)
I20260924 20:20:56.298581 29401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:48993
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:20:56.312126 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:56.423132 29597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:56.423396 29597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:56.423473 29597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:56.427457 29597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:56.427511 29597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:56.427613 29597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:20:56.432955 29597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:56.434016 29597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:56.435047 29597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:20:56.444962 29597 server_base.cc:1034] running on GCE node
W20260924 20:20:56.444973 29603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:56.444957 29605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:56.444957 29602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:56.445695 29597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:56.446424 29597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:56.447613 29597 hybrid_clock.cc:648] HybridClock initialized: now 1790281256447595 us; error 32 us; skew 500 ppm
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:56.450816 29597 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:20:56.451803 29597 webserver.cc:460] Webserver started at http://127.18.172.66:39819/ using document root <none> and password file <none>
I20260924 20:20:56.452391 29597 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:56.452485 29597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:56.452718 29597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:56.454810 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "3c9149e10d9b4db98009f68b8f51c31b"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "f0b42ae4b4f8cb56effedbfecc976a7b"
server_key_iv: "558b6485704eb9d751fbbae7e8dcc2be"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.455463 29597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "3c9149e10d9b4db98009f68b8f51c31b"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "f0b42ae4b4f8cb56effedbfecc976a7b"
server_key_iv: "558b6485704eb9d751fbbae7e8dcc2be"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.459252 29597 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:20:56.461699 29612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.462968 29597 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:56.463140 29597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "3c9149e10d9b4db98009f68b8f51c31b"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "f0b42ae4b4f8cb56effedbfecc976a7b"
server_key_iv: "558b6485704eb9d751fbbae7e8dcc2be"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.463249 29597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:56.490120 29597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:56.493187 29597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:56.493381 29597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:56.493989 29597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:56.494820 29597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:56.494866 29597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.494930 29597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:56.494964 29597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.504184 29597 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:41313
I20260924 20:20:56.504209 29725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:41313 every 8 connection(s)
I20260924 20:20:56.505084 29597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:20:56.508775 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29597
I20260924 20:20:56.508910 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:20:56.509198 19121 external_mini_cluster.cc:1428] Setting key da9e00ce9ed2e17cc5d4f1d4e6bd4051
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:20:56.520396 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.506825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52121 (local address 127.18.172.126:42823)
0924 20:20:56.506958 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.506961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.507820 (+ 859us) server_negotiation.cc:407] Connection header received
0924 20:20:56.509433 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.509436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.509498 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.509597 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:56.510890 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.511808 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.513223 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.513632 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.513765 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.516382 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:56.516390 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:56.516395 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:56.516434 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:56.518203 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.518715 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.518720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.518724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.518817 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.519193 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.519197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.519199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.519480 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:56.519594 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.520124 (+ 530us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.520262 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:20:56.521406 29728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.507223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:42823 (local address 127.18.172.66:52121)
0924 20:20:56.507662 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.507704 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.509092 (+ 1388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.509785 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.509795 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.510155 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:56.510735 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.510752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.511942 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.511944 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.513046 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.513056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.513805 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.513808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.513954 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.514622 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:56.514655 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:56.516199 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:56.518339 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.518343 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.518356 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.518608 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.518911 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.518913 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.518915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.519097 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.519587 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:56.519592 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:56.519828 (+ 236us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.520029 (+ 201us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.520327 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260924 20:20:56.522876 29726 heartbeater.cc:347] Connected to a master server at 127.18.172.126:42823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:20:56.523191 29726 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:56.523870 29726 heartbeater.cc:511] Master 127.18.172.126:42823 requested a full tablet report, sending...
I20260924 20:20:56.525009 29401 ts_manager.cc:194] Registered new tserver with Master: 3c9149e10d9b4db98009f68b8f51c31b (127.18.172.66:41313)
I20260924 20:20:56.525590 29401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:52121
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:20:56.540524 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:20:56.650161 29731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:20:56.650478 29731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:20:56.650554 29731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:20:56.654831 29731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:20:56.654929 29731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:20:56.655038 29731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:20:56.660486 29731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:42823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:20:56.661625 29731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:20:56.662659 29731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:20:56.672839 29737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:56.672926 29731 server_base.cc:1034] running on GCE node
W20260924 20:20:56.672811 29736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:20:56.672823 29739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:20:56.673533 29731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:20:56.674186 29731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:20:56.675375 29731 hybrid_clock.cc:648] HybridClock initialized: now 1790281256675355 us; error 32 us; skew 500 ppm
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:20:56.678400 29731 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:20:56.679497 29731 webserver.cc:460] Webserver started at http://127.18.172.67:34305/ using document root <none> and password file <none>
I20260924 20:20:56.680073 29731 fs_manager.cc:359] Metadata directory not provided
I20260924 20:20:56.680161 29731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:20:56.680394 29731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:20:56.682454 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "cb5b6834bb204a05b668d0542418821c"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "a50a97812460a09eaa581e63a93d297e"
server_key_iv: "5eac3c6edbb07a95b6950f90318d6df5"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.683081 29731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "cb5b6834bb204a05b668d0542418821c"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "a50a97812460a09eaa581e63a93d297e"
server_key_iv: "5eac3c6edbb07a95b6950f90318d6df5"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.686858 29731 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:20:56.689379 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.690621 29731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:20:56.690785 29731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "cb5b6834bb204a05b668d0542418821c"
format_stamp: "Formatted at 2026-09-24 20:20:56 on dist-test-slave-m1qq"
server_key: "a50a97812460a09eaa581e63a93d297e"
server_key_iv: "5eac3c6edbb07a95b6950f90318d6df5"
server_key_version: "encryptionkey@0"
I20260924 20:20:56.690898 29731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:20:56.703460 29731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:20:56.706523 29731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:20:56.706724 29731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:20:56.707350 29731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:20:56.708164 29731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:20:56.708237 29731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.708307 29731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:20:56.708366 29731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:20:56.718003 29731 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:40415
I20260924 20:20:56.718029 29862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:40415 every 8 connection(s)
I20260924 20:20:56.718909 29731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:20:56.726775 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 29731
I20260924 20:20:56.726923 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:20:56.727303 19121 external_mini_cluster.cc:1428] Setting key 8f20bdab0e4a8ab48072344983170354
Sep 24 20:20:56 dist-test-slave-m1qq krb5kdc[29356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281256, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:20:56.733946 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.720869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:36033 (local address 127.18.172.126:42823)
0924 20:20:56.721004 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.721008 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.721867 (+ 859us) server_negotiation.cc:407] Connection header received
0924 20:20:56.723397 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.723403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.723467 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.723554 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:56.724711 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.725271 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.726367 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.726668 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.726793 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.730010 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:56.730021 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:56.730023 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:56.730049 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:56.731702 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.732260 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.732263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.732266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.732351 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.732721 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.732726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.732728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.732928 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:56.733046 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.733590 (+ 544us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.733732 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:20:56.734926 29865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.721261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:42823 (local address 127.18.172.67:36033)
0924 20:20:56.721704 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.721751 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.723151 (+ 1400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.723733 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.723740 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.724096 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:20:56.724571 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.724584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.725381 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.725384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.726256 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.726265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.726817 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.726820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.727083 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.727806 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:20:56.727832 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:20:56.729828 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:20:56.731824 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.731830 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.731846 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.732128 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.732449 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:20:56.732454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:20:56.732456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:20:56.732612 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:20:56.733043 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:20:56.733050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:20:56.733327 (+ 277us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.733548 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.733916 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260924 20:20:56.736166 29863 heartbeater.cc:347] Connected to a master server at 127.18.172.126:42823
I20260924 20:20:56.736430 29863 heartbeater.cc:464] Registering TS with master...
I20260924 20:20:56.736943 29863 heartbeater.cc:511] Master 127.18.172.126:42823 requested a full tablet report, sending...
I20260924 20:20:56.737986 29401 ts_manager.cc:194] Registered new tserver with Master: cb5b6834bb204a05b668d0542418821c (127.18.172.67:40415)
I20260924 20:20:56.738555 29401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:36033
I20260924 20:20:56.742060 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:20:56.752760 29440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.744333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54742 (local address 127.18.172.126:42823)
0924 20:20:56.744481 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.744485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.744569 (+ 84us) server_negotiation.cc:407] Connection header received
0924 20:20:56.744672 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.744675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.744723 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.744788 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:20:56.745598 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.746119 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.747420 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.747655 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.747753 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.749149 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:20:56.749152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:20:56.749154 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:20:56.749179 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:20:56.750738 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.751371 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.751375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.751376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.751463 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:20:56.751734 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:20:56.751736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:20:56.751738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:20:56.751969 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:20:56.752071 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.752428 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.752537 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:20:56.755826 29401 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:20:56.759187 29401 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:20:56.774596 29874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.768465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:40419 (local address 127.0.0.1:57266)
0924 20:20:56.769380 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.769394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.769465 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.770407 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.770410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.770428 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:56.770529 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.770535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.771835 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.771839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.772893 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.772904 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.774185 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.774189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.774286 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.774309 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.774378 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.774442 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"thread_start_us":98,"threads_started":1}
I20260924 20:20:56.774860 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.768327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.18.172.66:41313)
0924 20:20:56.769102 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.769115 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.769139 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:20:56.769233 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.769238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.769458 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.769648 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:56.769944 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.770855 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.772619 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.773598 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.773711 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.773869 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.774535 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.774658 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":101,"threads_started":1}
I20260924 20:20:56.775092 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.768180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:41313 (local address 127.0.0.1:58906)
0924 20:20:56.768829 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.768900 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.769023 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.769630 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.769636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.769659 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:56.769814 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.769820 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.770982 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.770985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.772485 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.772496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.773935 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.773940 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.774049 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.774091 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.774207 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.774343 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260924 20:20:56.775332 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.768977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.18.172.65:40419)
0924 20:20:56.769994 (+ 1017us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.770011 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.770034 (+ 23us) server_negotiation.cc:407] Connection header received
0924 20:20:56.770108 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.770112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.770267 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.770431 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:56.770669 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.771704 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.773037 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.773880 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.773977 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.774110 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.775084 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.775191 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":60,"threads_started":1}
I20260924 20:20:56.777796 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.768932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:40415 (local address 127.0.0.1:38234)
0924 20:20:56.770676 (+ 1744us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.770694 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.770795 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.771315 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.771318 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.771337 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:56.771417 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.771422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.773727 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.773730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.774818 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.774828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.777396 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.777400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.777486 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.777509 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.777591 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.777673 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1678,"thread_start_us":90,"threads_started":1}
I20260924 20:20:56.778126 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.769400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.18.172.67:40415)
0924 20:20:56.769730 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.769742 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.770787 (+ 1045us) server_negotiation.cc:407] Connection header received
0924 20:20:56.770944 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.770953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.771160 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.771315 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:56.771510 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.773558 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.774955 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.777067 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.777198 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.777330 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.777645 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.777744 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":138,"threads_started":1}
I20260924 20:20:56.778491 29526 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 (DEFAULT_TABLE table=test-table [id=c1df166bdc924e94bb893d7acfee7da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:56.778496 29660 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 (DEFAULT_TABLE table=test-table [id=c1df166bdc924e94bb893d7acfee7da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:56.779513 29660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc584f77f9541a2a4850ab084aafeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:56.779541 29526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc584f77f9541a2a4850ab084aafeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:56.780947 29795 tablet_service.cc:1467] Processing CreateTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 (DEFAULT_TABLE table=test-table [id=c1df166bdc924e94bb893d7acfee7da0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:20:56.781984 29795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fc584f77f9541a2a4850ab084aafeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:20:56.785840 29880 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Bootstrap starting.
I20260924 20:20:56.787202 29881 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: Bootstrap starting.
I20260924 20:20:56.788381 29880 tablet_bootstrap.cc:654] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:56.789201 29882 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Bootstrap starting.
I20260924 20:20:56.789224 29880 log.cc:824] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:56.789261 29881 tablet_bootstrap.cc:654] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:56.790467 29881 log.cc:824] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:56.791388 29882 tablet_bootstrap.cc:654] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Neither blocks nor log segments found. Creating new log.
I20260924 20:20:56.791474 29880 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: No bootstrap required, opened a new log
I20260924 20:20:56.791672 29880 ts_tablet_manager.cc:1397] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:20:56.792171 29881 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: No bootstrap required, opened a new log
I20260924 20:20:56.792292 29881 ts_tablet_manager.cc:1397] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:20:56.792567 29882 log.cc:824] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:20:56.794394 29882 tablet_bootstrap.cc:492] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: No bootstrap required, opened a new log
I20260924 20:20:56.794546 29882 ts_tablet_manager.cc:1397] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:20:56.795567 29881 raft_consensus.cc:348] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.795781 29881 raft_consensus.cc:374] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:56.795624 29880 raft_consensus.cc:348] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.795814 29881 raft_consensus.cc:729] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4115ffb99140768f82ef4c34c3ddf1, State: Initialized, Role: FOLLOWER
I20260924 20:20:56.795856 29880 raft_consensus.cc:374] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:56.795904 29880 raft_consensus.cc:729] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9149e10d9b4db98009f68b8f51c31b, State: Initialized, Role: FOLLOWER
I20260924 20:20:56.796216 29881 consensus_queue.cc:260] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.796307 29880 consensus_queue.cc:260] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.796748 29592 heartbeater.cc:503] Master 127.18.172.126:42823 was elected leader, sending a full tablet report...
I20260924 20:20:56.797122 29881 ts_tablet_manager.cc:1428] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:20:56.797221 29880 ts_tablet_manager.cc:1428] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:20:56.797530 29726 heartbeater.cc:503] Master 127.18.172.126:42823 was elected leader, sending a full tablet report...
I20260924 20:20:56.798358 29882 raft_consensus.cc:348] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.798669 29882 raft_consensus.cc:374] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:20:56.798749 29882 raft_consensus.cc:729] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5b6834bb204a05b668d0542418821c, State: Initialized, Role: FOLLOWER
I20260924 20:20:56.799263 29882 consensus_queue.cc:260] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.800000 29863 heartbeater.cc:503] Master 127.18.172.126:42823 was elected leader, sending a full tablet report...
I20260924 20:20:56.800369 29882 ts_tablet_manager.cc:1428] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:20:56.870875 29886 raft_consensus.cc:482] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:20:56.871160 29886 raft_consensus.cc:504] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.872291 29886 leader_election.cc:288] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9149e10d9b4db98009f68b8f51c31b (127.18.172.66:41313), cb5b6834bb204a05b668d0542418821c (127.18.172.67:40415)
I20260924 20:20:56.878392 29890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.872698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:41313 (local address 127.18.172.65:43473)
0924 20:20:56.873033 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.873045 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.873194 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.873452 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.873454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.873470 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:56.873681 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.873691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.874717 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.874722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.876703 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.876717 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.877823 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.877827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.877978 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.878001 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.878133 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.878216 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":124,"spinlock_wait_cycles":55296,"thread_start_us":105,"threads_started":1}
I20260924 20:20:56.878458 29875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.872786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:43473 (local address 127.18.172.66:41313)
0924 20:20:56.872919 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.872924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.873129 (+ 205us) server_negotiation.cc:407] Connection header received
0924 20:20:56.873301 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.873306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.873356 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.873440 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:56.873793 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.874599 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.876844 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.877678 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.877823 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.877881 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.878189 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.878296 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:20:56.878484 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.872698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46299 (local address 127.18.172.67:40415)
0924 20:20:56.872884 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.872889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.873120 (+ 231us) server_negotiation.cc:407] Connection header received
0924 20:20:56.873334 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.873338 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.873391 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.873579 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:20:56.873849 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.874595 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.876898 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.877690 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.877842 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.877914 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.878273 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.878354 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 20:20:56.879160 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.872609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:40415 (local address 127.18.172.65:46299)
0924 20:20:56.873022 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.873034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:20:56.873166 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:20:56.873509 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:20:56.873512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:20:56.873554 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:20:56.873708 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.873724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.874749 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.874755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.876707 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:20:56.876723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.877986 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.877993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:20:56.878112 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.878137 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:20:56.878887 (+ 750us) client_negotiation.cc:240] Negotiation successful
0924 20:20:56.878968 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":265,"spinlock_wait_cycles":1280,"thread_start_us":154,"threads_started":1}
I20260924 20:20:56.879819 29680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc584f77f9541a2a4850ab084aafeb7" candidate_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9149e10d9b4db98009f68b8f51c31b" is_pre_election: true
I20260924 20:20:56.880105 29816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc584f77f9541a2a4850ab084aafeb7" candidate_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5b6834bb204a05b668d0542418821c" is_pre_election: true
I20260924 20:20:56.880170 29680 raft_consensus.cc:2454] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4115ffb99140768f82ef4c34c3ddf1 in term 0.
I20260924 20:20:56.880342 29816 raft_consensus.cc:2454] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c4115ffb99140768f82ef4c34c3ddf1 in term 0.
I20260924 20:20:56.880755 29482 leader_election.cc:302] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c4115ffb99140768f82ef4c34c3ddf1, 3c9149e10d9b4db98009f68b8f51c31b, cb5b6834bb204a05b668d0542418821c; no voters:
I20260924 20:20:56.881436 29886 raft_consensus.cc:2781] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:20:56.881567 29886 raft_consensus.cc:482] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:20:56.881630 29886 raft_consensus.cc:3037] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:56.883134 29886 raft_consensus.cc:504] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.883567 29886 leader_election.cc:288] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [CANDIDATE]: Term 1 election: Requested vote from peers 3c9149e10d9b4db98009f68b8f51c31b (127.18.172.66:41313), cb5b6834bb204a05b668d0542418821c (127.18.172.67:40415)
I20260924 20:20:56.883939 29680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc584f77f9541a2a4850ab084aafeb7" candidate_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9149e10d9b4db98009f68b8f51c31b"
I20260924 20:20:56.884063 29680 raft_consensus.cc:3037] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:56.884120 29816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fc584f77f9541a2a4850ab084aafeb7" candidate_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb5b6834bb204a05b668d0542418821c"
I20260924 20:20:56.884224 29816 raft_consensus.cc:3037] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:20:56.885078 29680 raft_consensus.cc:2454] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4115ffb99140768f82ef4c34c3ddf1 in term 1.
I20260924 20:20:56.885138 29816 raft_consensus.cc:2454] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c4115ffb99140768f82ef4c34c3ddf1 in term 1.
I20260924 20:20:56.885495 29480 leader_election.cc:302] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c4115ffb99140768f82ef4c34c3ddf1, 3c9149e10d9b4db98009f68b8f51c31b, cb5b6834bb204a05b668d0542418821c; no voters:
I20260924 20:20:56.885681 29886 raft_consensus.cc:2781] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:20:56.885950 29886 raft_consensus.cc:686] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [term 1 LEADER]: Becoming Leader. State: Replica: 2c4115ffb99140768f82ef4c34c3ddf1, State: Running, Role: LEADER
I20260924 20:20:56.886312 29886 consensus_queue.cc:237] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } }
I20260924 20:20:56.889428 29401 catalog_manager.cc:5495] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c4115ffb99140768f82ef4c34c3ddf1 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c4115ffb99140768f82ef4c34c3ddf1" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 40419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 } health_report { overall_health: UNKNOWN } } }
I20260924 20:20:56.929302 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.926090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57272 (local address 127.18.172.65:40419)
0924 20:20:56.926217 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:20:56.926221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:20:56.926233 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:20:56.926325 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:20:56.926327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:20:56.926379 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:20:56.926471 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:20:56.926686 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.927198 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.928024 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:20:56.928225 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:20:56.928310 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:20:56.928579 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:20:56.928993 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:20:56.929082 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:20:56.929131 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 20:20:56.929180 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:20:56.935628 29680 raft_consensus.cc:1261] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 1 FOLLOWER]: Refusing update from remote peer 2c4115ffb99140768f82ef4c34c3ddf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:56.935628 29816 raft_consensus.cc:1261] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 1 FOLLOWER]: Refusing update from remote peer 2c4115ffb99140768f82ef4c34c3ddf1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:20:56.936465 29886 consensus_queue.cc:1035] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9149e10d9b4db98009f68b8f51c31b" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 41313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:56.936676 29891 consensus_queue.cc:1035] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb5b6834bb204a05b668d0542418821c" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:20:56.944618 29894 mvcc.cc:205] Tried to move back new op lower bound from 7332992028400889856 to 7332992028208922624. Current Snapshot: MvccSnapshot[applied={T|T < 7332992028400889856}]
I20260924 20:20:56.946916 29895 mvcc.cc:205] Tried to move back new op lower bound from 7332992028400889856 to 7332992028208922624. Current Snapshot: MvccSnapshot[applied={T|T < 7332992028400889856}]
I20260924 20:20:56.953524 29401 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:56.953732 29401 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:20:56.956333 29401 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ff149b4e83f4e6ba6a076593ea7f2c8: Sending DeleteTablet for 3 replicas of tablet 1fc584f77f9541a2a4850ab084aafeb7
I20260924 20:20:56.957140 29526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:56 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:57266
I20260924 20:20:56.957223 29660 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:56 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:58906
I20260924 20:20:56.957284 29795 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fc584f77f9541a2a4850ab084aafeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:20:56 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:38234
I20260924 20:20:56.957739 29904 tablet_replica.cc:330] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: stopping tablet replica
I20260924 20:20:56.957798 29903 tablet_replica.cc:330] T 1fc584f77f9541a2a4850ab084aafeb7 P 2c4115ffb99140768f82ef4c34c3ddf1: stopping tablet replica
I20260924 20:20:56.957964 29905 tablet_replica.cc:330] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: stopping tablet replica
I20260924 20:20:56.958282 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29463
I20260924 20:20:56.959542 29904 raft_consensus.cc:2229] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:56.959829 29905 raft_consensus.cc:2229] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:20:56.959906 29904 raft_consensus.cc:2258] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:56.960134 29905 raft_consensus.cc:2258] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:20:56.961505 29905 ts_tablet_manager.cc:1905] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:56.961568 29904 ts_tablet_manager.cc:1905] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:20:56.963716 29905 ts_tablet_manager.cc:1918] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:56.963835 29905 log.cc:1196] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/1fc584f77f9541a2a4850ab084aafeb7
I20260924 20:20:56.964170 29905 ts_tablet_manager.cc:1939] T 1fc584f77f9541a2a4850ab084aafeb7 P cb5b6834bb204a05b668d0542418821c: Deleting consensus metadata
I20260924 20:20:56.964931 29904 ts_tablet_manager.cc:1918] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:20:56.965030 29904 log.cc:1196] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/1fc584f77f9541a2a4850ab084aafeb7
I20260924 20:20:56.965353 29904 ts_tablet_manager.cc:1939] T 1fc584f77f9541a2a4850ab084aafeb7 P 3c9149e10d9b4db98009f68b8f51c31b: Deleting consensus metadata
I20260924 20:20:56.965391 29386 catalog_manager.cc:4841] TS cb5b6834bb204a05b668d0542418821c (127.18.172.67:40415): tablet 1fc584f77f9541a2a4850ab084aafeb7 (table test-table [id=c1df166bdc924e94bb893d7acfee7da0]) successfully deleted
I20260924 20:20:56.966609 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29597
W20260924 20:20:56.966625 29385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:40419 (error 108)
I20260924 20:20:56.966773 29388 catalog_manager.cc:4841] TS 3c9149e10d9b4db98009f68b8f51c31b (127.18.172.66:41313): tablet 1fc584f77f9541a2a4850ab084aafeb7 (table test-table [id=c1df166bdc924e94bb893d7acfee7da0]) successfully deleted
I20260924 20:20:56.968613 29877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:20:56.968341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:40419 (local address 127.0.0.1:57284)
0924 20:20:56.968470 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:20:56.968542 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:40419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:20:56.968772 29385 catalog_manager.cc:4584] TS 2c4115ffb99140768f82ef4c34c3ddf1 (127.18.172.65:40419): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1fc584f77f9541a2a4850ab084aafeb7: Network error: Client connection negotiation failed: client connection to 127.18.172.65:40419: connect: Connection refused (error 111)
I20260924 20:20:56.973414 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29731
I20260924 20:20:56.978752 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 29369
2026-09-24T20:20:56Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2370 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:20:57 dist-test-slave-m1qq krb5kdc[29909](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:20:57 dist-test-slave-m1qq krb5kdc[29909](info): set up 2 sockets
Sep 24 20:20:57 dist-test-slave-m1qq krb5kdc[29909](info): commencing operation
krb5kdc: starting...
W20260924 20:20:58.029229 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:20:58 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:20:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:20:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangeradmin_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangeradmin_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangerlookup_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangerlookup_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
I20260924 20:20:58.201826 19121 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQK3utR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/postgres -l logfile start
2026-09-24 20:21:00.908 UTC [29940] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:21:00.908 UTC [29940] LOG: listening on IPv4 address "127.18.172.84", port 47769
2026-09-24 20:21:00.912 UTC [29940] LOG: listening on Unix socket "/tmp/.s.PGSQL.47769"
2026-09-24 20:21:00.934 UTC [29943] LOG: database system was shut down at 2026-09-24 20:20:59 UTC
2026-09-24 20:21:00.938 UTC [29940] LOG: database system is ready to accept connections
I20260924 20:21:01.916990 19121 mini_postgres.cc:96] Postgres bound to 47769
2026-09-24 20:21:01.923 UTC [29951] FATAL: database "slave" does not exist
127.18.172.84:47769 - accepting connections
I20260924 20:21:01.924305 19121 mini_ranger.cc:161] Starting Ranger...
I20260924 20:21:01.939121 19121 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:21:02.500648 19121 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:21:02.500735 19121 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin
2026-09-24 20:21:02,797 [I] DB FLAVOR :POSTGRES
2026-09-24 20:21:02,797 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:21:02,797 [I] Checking connection..
2026-09-24 20:21:02,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:21:03,068 [I] Checking connection passed.
2026-09-24 20:21:03,068 [I] --------- Verifying version history table ---------
2026-09-24 20:21:03,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:03,336 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:21:03,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:03,566 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:21:03,566 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:21:03,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:21:03.806 UTC [30016] WARNING: there is no transaction in progress
2026-09-24 20:21:03,813 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:21:03,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:04,045 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:21:04,045 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:21:04,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:21:04,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:21:04,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m1qq.c.gcp-upstream.internal','N') ;"
2026-09-24 20:21:04,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:21:05,049 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:21:05,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:21:05,283 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:21:05,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:21:05,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:21:05,784 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:21:05,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:21:06.778 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.797 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.814 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.832 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.849 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.872 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.931 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.939 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.957 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.969 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.981 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:06.996 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:07.006 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:07.014 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:07.021 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:07.030 UTC [30124] WARNING: there is no transaction in progress
2026-09-24 20:21:07,760 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:21:07,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:21:08,001 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:21:08,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:21:08,239 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:21:08,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:21:08,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:21:08,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:21:08,726 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:21:08,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m1qq.c.gcp-upstream.internal';"
2026-09-24 20:21:09,225 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:21:09,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:21:09,483 [I] DB_PATCHES have already been applied
I20260924 20:21:09.488993 19121 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQK3utR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:21:09.489060 19121 mini_ranger.cc:193] Using host: 127.18.172.84
I20260924 20:21:09.492538 19121 mini_ranger.cc:227] Ranger admin URL: http://127.18.172.84:56179
Sep 24, 2026 8:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:21:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:21:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 20:21:10.592314 19121 mini_ranger.cc:160] Time spent starting Ranger: real 8.668s user 0.001s sys 0.009s
Sep 24, 2026 8:21:31 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:21:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 8:21:31 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:21:31 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:21:32 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:21:32.929695 19121 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangerkms_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/rangerkms_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.172.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.172.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 20:21:33 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281293, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 20:21:33.019634 19121 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 20:21:33.029258 19121 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 20:21:33.998306 19121 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 20:21:33.998407 19121 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms
I20260924 20:21:33.998510 19121 mini_ranger_kms.cc:227] Using postgres at 127.18.172.84:47769
2026-09-24 20:21:34,184 [I] DB FLAVOR :POSTGRES
2026-09-24 20:21:34,184 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:21:34,184 [I] Checking connection
2026-09-24 20:21:34,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 20:21:34,424 [I] connection success
2026-09-24 20:21:34,424 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 20:21:34,424 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 20:21:34,425 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 20:21:34,679 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 20:21:34,680 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:21:34,680 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 20:21:34,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:47769/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 20:21:34,959 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 20:21:35.446307 19121 mini_ranger_kms.cc:310] Created kms service
I20260924 20:21:35.646852 19121 mini_ranger_kms.cc:326] Created kudu user
I20260924 20:21:35.691179 19121 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 20:21:35 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281293, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:36.110314 19121 mini_ranger_kms.cc:384] Added ranger policy
I20260924 20:21:36.110488 19121 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 20:21:36.110514 19121 mini_ranger_kms.cc:240] Using host: 127.18.172.84
I20260924 20:21:36.114100 19121 mini_ranger_kms.cc:276] Ranger KMS PID: 30306
I20260924 20:21:36.114248 19121 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.172.84:44727
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 8:21:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 20:21:38.388249 19121 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.369s user 0.007s sys 0.163s
I20260924 20:21:38.388394 19121 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 20:21:38.388540 19121 mini_ranger_kms.cc:401] 127.18.172.84:44727/kms/v1/keys
Sep 24 20:21:38 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 20:21:38 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
Sep 24 20:21:39 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281298, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
Sep 24 20:21:41 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:21:41.531359 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34059
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34059
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:44727/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:21:41.644889 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:21:41.645200 30357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:21:41.645283 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:21:41.649385 30357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:21:41.649483 30357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:21:41.649523 30357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:21:41.649580 30357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:21:41.649619 30357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:21:41.655166 30357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.172.84:44727/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34059
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:21:41.656275 30357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:21:41.657327 30357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:21:41.667450 30357 server_base.cc:1034] running on GCE node
W20260924 20:21:41.667375 30362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:41.667377 30363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:41.667695 30365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:41.668282 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:21:41.669446 30357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:21:41.670696 30357 hybrid_clock.cc:648] HybridClock initialized: now 1790281301670681 us; error 44 us; skew 500 ppm
Sep 24 20:21:41 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281301, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:21:41.677357 30357 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:21:41.678570 30357 webserver.cc:460] Webserver started at http://127.18.172.126:41483/ using document root <none> and password file <none>
I20260924 20:21:41.679309 30357 fs_manager.cc:359] Metadata directory not provided
I20260924 20:21:41.679387 30357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:21:41.679621 30357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:21:41 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281301, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:41.726933 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "34550353426d4bbd9b94b7d5a077b4a3"
format_stamp: "Formatted at 2026-09-24 20:21:41 on dist-test-slave-m1qq"
server_key: "3fe7efdeceee3730c32e7ae57b32964c"
server_key_iv: "6e20f625edda7cf5dfb6b3f3e2a50800"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:41.727659 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "34550353426d4bbd9b94b7d5a077b4a3"
format_stamp: "Formatted at 2026-09-24 20:21:41 on dist-test-slave-m1qq"
server_key: "3fe7efdeceee3730c32e7ae57b32964c"
server_key_iv: "6e20f625edda7cf5dfb6b3f3e2a50800"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:41.731887 30357 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:21:41.749109 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:41.750648 30357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:21:41.750816 30357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "34550353426d4bbd9b94b7d5a077b4a3"
format_stamp: "Formatted at 2026-09-24 20:21:41 on dist-test-slave-m1qq"
server_key: "3fe7efdeceee3730c32e7ae57b32964c"
server_key_iv: "6e20f625edda7cf5dfb6b3f3e2a50800"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:41.750962 30357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:21:41.766070 30357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:21:41.769184 30357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:21:41.769394 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:21:41.777328 30357 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:34059
I20260924 20:21:41.777379 30425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:34059 every 8 connection(s)
I20260924 20:21:41.778364 30357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:21:41.779162 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 30357
I20260924 20:21:41.779309 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:21:41.779600 19121 external_mini_cluster.cc:1428] Setting key 15cdc5f4e4c41d1ae90450cf5118bc66
I20260924 20:21:41.781883 30426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 20:21:41 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:21:41.789258 30426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: Bootstrap starting.
I20260924 20:21:41.792030 30426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: Neither blocks nor log segments found. Creating new log.
I20260924 20:21:41.793385 30426 log.cc:824] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:21:41.794493 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:41.781051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45058 (local address 127.18.172.126:34059)
0924 20:21:41.781470 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:41.781491 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:41.781733 (+ 242us) server_negotiation.cc:407] Connection header received
0924 20:21:41.782646 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:41.782666 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:41.783055 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:41.783413 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:21:41.784179 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:41.785234 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:41.786116 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:41.786608 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:41.786787 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:41.789400 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:21:41.789408 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:21:41.789426 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:21:41.789464 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:21:41.791263 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:41.791724 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:41.791735 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:41.791740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:41.791842 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:41.792205 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:41.792209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:41.792211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:41.792623 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:21:41.792834 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:41.793139 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 20:21:41.793602 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":138,"threads_started":1}
I20260924 20:21:41.795719 30426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: No bootstrap required, opened a new log
I20260924 20:21:41.799628 30426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } }
I20260924 20:21:41.799896 30426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:21:41.799950 30426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34550353426d4bbd9b94b7d5a077b4a3, State: Initialized, Role: FOLLOWER
I20260924 20:21:41.800400 30426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } }
I20260924 20:21:41.800539 30426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:21:41.800597 30426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:21:41.800685 30426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:21:41.801738 30426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } }
I20260924 20:21:41.802084 30426 leader_election.cc:302] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34550353426d4bbd9b94b7d5a077b4a3; no voters:
I20260924 20:21:41.802423 30426 leader_election.cc:288] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:21:41.802536 30431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:21:41.802798 30431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [term 1 LEADER]: Becoming Leader. State: Replica: 34550353426d4bbd9b94b7d5a077b4a3, State: Running, Role: LEADER
I20260924 20:21:41.803164 30431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } }
I20260924 20:21:41.803699 30426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:21:41.804839 30433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34550353426d4bbd9b94b7d5a077b4a3. Latest consensus state: current_term: 1 leader_uuid: "34550353426d4bbd9b94b7d5a077b4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } } }
I20260924 20:21:41.804991 30433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:21:41.805385 30441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:21:41.805325 30432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34550353426d4bbd9b94b7d5a077b4a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34550353426d4bbd9b94b7d5a077b4a3" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34059 } } }
I20260924 20:21:41.805482 30432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:21:41.805796 30426 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:21:41.812729 30441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:21:41.820756 30441 catalog_manager.cc:1347] Generated new cluster ID: 0e67380d529f419b8e024e4a526b0a03
I20260924 20:21:41.820839 30441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:21:41.835341 30441 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:21:41.836856 30441 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:21:41.854091 30441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: Generated new TSK 0
I20260924 20:21:41.855211 30441 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:21:42.610273 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:44727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:21:42.757223 30471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:21:42.757527 30471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:21:42.757637 30471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:21:42.761838 30471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:21:42.761950 30471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:21:42.762056 30471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:21:42.767787 30471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.172.84:44727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:21:42.769147 30471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:21:42.770417 30471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:21:42.781239 30479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:42.781239 30478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:42.781339 30481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:42.782645 30471 server_base.cc:1034] running on GCE node
I20260924 20:21:42.783200 30471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:21:42.783978 30471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:21:42.785197 30471 hybrid_clock.cc:648] HybridClock initialized: now 1790281302785135 us; error 85 us; skew 500 ppm
Sep 24 20:21:42 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281302, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:21:42.788764 30471 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:21:42.789853 30471 webserver.cc:460] Webserver started at http://127.18.172.65:39015/ using document root <none> and password file <none>
I20260924 20:21:42.790598 30471 fs_manager.cc:359] Metadata directory not provided
I20260924 20:21:42.790705 30471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:21:42.790933 30471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:21:42 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281302, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:42.822548 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "b562ef9b0ac44a18b1048ef9bced021e"
format_stamp: "Formatted at 2026-09-24 20:21:42 on dist-test-slave-m1qq"
server_key: "b5a2b8ce88bee026c07912c5cbce75b8"
server_key_iv: "b0c4023cb5e9c014fea2b16281c6b149"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:42.831601 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "b562ef9b0ac44a18b1048ef9bced021e"
format_stamp: "Formatted at 2026-09-24 20:21:42 on dist-test-slave-m1qq"
server_key: "b5a2b8ce88bee026c07912c5cbce75b8"
server_key_iv: "b0c4023cb5e9c014fea2b16281c6b149"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:42.838034 30471 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 20:21:42.889643 30490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:42.891515 30471 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:21:42.891645 30471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "b562ef9b0ac44a18b1048ef9bced021e"
format_stamp: "Formatted at 2026-09-24 20:21:42 on dist-test-slave-m1qq"
server_key: "b5a2b8ce88bee026c07912c5cbce75b8"
server_key_iv: "b0c4023cb5e9c014fea2b16281c6b149"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:42.891779 30471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:21:42.931732 30471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:21:42.935575 30471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:21:42.935850 30471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:21:42.936627 30471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:21:42.937935 30471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:21:42.938054 30471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:42.938175 30471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:21:42.938251 30471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:42.952113 30471 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:34441
I20260924 20:21:42.953413 30471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:21:42.955163 30604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:34441 every 8 connection(s)
I20260924 20:21:42.960837 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 30471
I20260924 20:21:42.961005 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:21:42.961350 19121 external_mini_cluster.cc:1428] Setting key 9f8892e4a294ca0cea5338efe1e45f92
Sep 24 20:21:42 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281302, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:21:42.975153 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:42.957509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:40161 (local address 127.18.172.126:34059)
0924 20:21:42.957776 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:42.957781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:42.958906 (+ 1125us) server_negotiation.cc:407] Connection header received
0924 20:21:42.960195 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:42.960199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:42.960256 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:42.960335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:21:42.964300 (+ 3965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:42.965342 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:42.967091 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:42.967454 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:42.967534 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:42.970632 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:21:42.970636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:21:42.970638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:21:42.970665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:21:42.972552 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:42.973170 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:42.973173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:42.973175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:42.973265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:42.973708 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:42.973712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:42.973714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:42.973997 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:21:42.974100 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:42.974806 (+ 706us) server_negotiation.cc:299] Negotiation successful
0924 20:21:42.974936 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":74,"threads_started":1}
I20260924 20:21:42.977216 30608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:42.958118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34059 (local address 127.18.172.65:40161)
0924 20:21:42.958729 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:42.958793 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:42.959933 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:42.960775 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:42.960786 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:42.961298 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:21:42.964050 (+ 2752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:42.964074 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:42.965496 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:42.965499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:42.966920 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:42.966931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:42.967744 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:42.967748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:42.967987 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:42.968661 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:21:42.968683 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:21:42.970436 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:21:42.972686 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:42.972692 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:42.972711 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:42.973055 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:42.973397 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:42.973400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:42.973402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:42.973586 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:42.974173 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:21:42.974179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:21:42.974521 (+ 342us) client_negotiation.cc:769] Sending connection context
0924 20:21:42.975634 (+ 1113us) client_negotiation.cc:240] Negotiation successful
0924 20:21:42.976017 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":164,"threads_started":1}
I20260924 20:21:42.978945 30605 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34059
I20260924 20:21:42.979233 30605 heartbeater.cc:464] Registering TS with master...
I20260924 20:21:42.979800 30605 heartbeater.cc:511] Master 127.18.172.126:34059 requested a full tablet report, sending...
I20260924 20:21:42.981669 30389 ts_manager.cc:194] Registered new tserver with Master: b562ef9b0ac44a18b1048ef9bced021e (127.18.172.65:34441)
I20260924 20:21:42.983310 30389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:40161
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:21:43.050478 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:44727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:21:43.206631 30611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:21:43.207170 30611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:21:43.207365 30611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:21:43.213506 30611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:21:43.213580 30611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:21:43.213670 30611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:21:43.221710 30611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.172.84:44727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:21:43.223165 30611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:21:43.224501 30611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:21:43.238281 30618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:43.238725 30611 server_base.cc:1034] running on GCE node
W20260924 20:21:43.239094 30617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:43.239356 30620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:43.239915 30611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:21:43.240722 30611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:21:43.241885 30611 hybrid_clock.cc:648] HybridClock initialized: now 1790281303241817 us; error 84 us; skew 500 ppm
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:21:43.245697 30611 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:21:43.246815 30611 webserver.cc:460] Webserver started at http://127.18.172.66:43809/ using document root <none> and password file <none>
I20260924 20:21:43.247577 30611 fs_manager.cc:359] Metadata directory not provided
I20260924 20:21:43.247679 30611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:21:43.247956 30611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:43.270076 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "7f0b849c02724bef82fa7348afc35b6e"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "8bf134379fffda99051f5b0c37da794c"
server_key_iv: "266e15bcd466459386a16a9f21fd3bca"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.270898 30611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "7f0b849c02724bef82fa7348afc35b6e"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "8bf134379fffda99051f5b0c37da794c"
server_key_iv: "266e15bcd466459386a16a9f21fd3bca"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.277222 30611 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 20:21:43.302239 30627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.304340 30611 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260924 20:21:43.304525 30611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "7f0b849c02724bef82fa7348afc35b6e"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "8bf134379fffda99051f5b0c37da794c"
server_key_iv: "266e15bcd466459386a16a9f21fd3bca"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.304672 30611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:21:43.321476 30611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:21:43.325560 30611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:21:43.325802 30611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:21:43.326676 30611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:21:43.327973 30611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:21:43.328047 30611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.328114 30611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:21:43.328152 30611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.338960 30611 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:40647
I20260924 20:21:43.340147 30611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:21:43.348272 30742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:40647 every 8 connection(s)
I20260924 20:21:43.350481 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 30611
I20260924 20:21:43.350597 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:21:43.350907 19121 external_mini_cluster.cc:1428] Setting key a1db1e1db5d5f0b32f3571261df05366
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:21:43.375608 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.350410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:35011 (local address 127.18.172.126:34059)
0924 20:21:43.350546 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.350550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.351767 (+ 1217us) server_negotiation.cc:407] Connection header received
0924 20:21:43.355234 (+ 3467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.355239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.355298 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.355354 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:21:43.357267 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.358104 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.359913 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.360280 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.360363 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.364242 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:21:43.364247 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:21:43.364250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:21:43.364282 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:21:43.366164 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.371171 (+ 5007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.371177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.371179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.371293 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.371869 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.371873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.371875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.372175 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:21:43.372265 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.375226 (+ 2961us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.375400 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:21:43.378911 30745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.349318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34059 (local address 127.18.172.66:35011)
0924 20:21:43.351609 (+ 2291us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.351672 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.354929 (+ 3257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.355906 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.355917 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.356289 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:21:43.357079 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.357101 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.358274 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.358278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.359766 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.359777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.360604 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.360608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.360818 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.362128 (+ 1310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:21:43.362153 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:21:43.364011 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:21:43.366306 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:43.366315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:43.366338 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:43.366731 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:43.371495 (+ 4764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:43.371499 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:43.371501 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:43.371740 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:43.374420 (+ 2680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:21:43.374429 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:21:43.374851 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.377278 (+ 2427us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.377723 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2030,"thread_start_us":162,"threads_started":1}
I20260924 20:21:43.380586 30743 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34059
I20260924 20:21:43.380851 30743 heartbeater.cc:464] Registering TS with master...
I20260924 20:21:43.381507 30743 heartbeater.cc:511] Master 127.18.172.126:34059 requested a full tablet report, sending...
I20260924 20:21:43.382805 30389 ts_manager.cc:194] Registered new tserver with Master: 7f0b849c02724bef82fa7348afc35b6e (127.18.172.66:40647)
I20260924 20:21:43.383638 30389 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:35011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:21:43.408571 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:44727/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
W20260924 20:21:43.601655 30748 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:21:43.602048 30748 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:21:43.602155 30748 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:21:43.608453 30748 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:21:43.608606 30748 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:21:43.608741 30748 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:21:43.618069 30748 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.172.84:44727/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30748
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:21:43.619720 30748 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:21:43.621276 30748 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:21:43.635360 30754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:43.635747 30748 server_base.cc:1034] running on GCE node
W20260924 20:21:43.636369 30756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:21:43.639187 30753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:21:43.639784 30748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:21:43.640609 30748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:21:43.644009 30748 hybrid_clock.cc:648] HybridClock initialized: now 1790281303643930 us; error 87 us; skew 500 ppm
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:21:43.647704 30748 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:21:43.648985 30748 webserver.cc:460] Webserver started at http://127.18.172.67:38815/ using document root <none> and password file <none>
I20260924 20:21:43.649724 30748 fs_manager.cc:359] Metadata directory not provided
I20260924 20:21:43.649838 30748 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:21:43.650055 30748 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:43.666409 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "69330b0b9b1b47afb2451a68ce90b3fd"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "e1a72fa7d4796d64bd262548829ffcbe"
server_key_iv: "7f4a355845d89408320d930f706379a9"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.668378 30748 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "69330b0b9b1b47afb2451a68ce90b3fd"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "e1a72fa7d4796d64bd262548829ffcbe"
server_key_iv: "7f4a355845d89408320d930f706379a9"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.674489 30748 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 20:21:43.696547 30763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.698253 30748 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260924 20:21:43.698426 30748 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "69330b0b9b1b47afb2451a68ce90b3fd"
format_stamp: "Formatted at 2026-09-24 20:21:43 on dist-test-slave-m1qq"
server_key: "e1a72fa7d4796d64bd262548829ffcbe"
server_key_iv: "7f4a355845d89408320d930f706379a9"
server_key_version: "kuduclusterkey@0"
I20260924 20:21:43.698573 30748 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:21:43.731104 30748 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:21:43.734360 30748 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:21:43.734581 30748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:21:43.735321 30748 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:21:43.736276 30748 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:21:43.736347 30748 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.736446 30748 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:21:43.736492 30748 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:21:43.748015 30748 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:44651
I20260924 20:21:43.748966 30748 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:21:43.749957 30877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:44651 every 8 connection(s)
I20260924 20:21:43.751631 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 30748
I20260924 20:21:43.751772 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:21:43.752082 19121 external_mini_cluster.cc:1428] Setting key cb8d058dfe53474e970c0f62a8b5d694
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:21:43.763751 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.750930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:54577 (local address 127.18.172.126:34059)
0924 20:21:43.751168 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.751173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.752870 (+ 1697us) server_negotiation.cc:407] Connection header received
0924 20:21:43.754057 (+ 1187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.754060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.754115 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.754169 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:21:43.755620 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.756214 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.757411 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.757650 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.757771 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.760147 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:21:43.760153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:21:43.760160 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:21:43.760185 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:21:43.761635 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.762153 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.762157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.762158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.762238 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.762590 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.762593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.762595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.762775 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:21:43.762886 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.763460 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.763578 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:21:43.764981 30880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.751486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34059 (local address 127.18.172.67:54577)
0924 20:21:43.752708 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.752764 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.753827 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.754567 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.754573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.754846 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:21:43.755478 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.755494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.756375 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.756379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.757282 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.757291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.757814 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.757818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.757987 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.758581 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:21:43.758597 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:21:43.759984 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:21:43.761769 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:43.761775 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:43.761790 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:43.762047 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:43.762334 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:21:43.762339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:21:43.762341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:21:43.762491 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:21:43.762898 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:21:43.762906 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:21:43.763220 (+ 314us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.763410 (+ 190us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.763715 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":133,"threads_started":1}
I20260924 20:21:43.766394 30878 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34059
I20260924 20:21:43.766644 30878 heartbeater.cc:464] Registering TS with master...
I20260924 20:21:43.767194 30878 heartbeater.cc:511] Master 127.18.172.126:34059 requested a full tablet report, sending...
I20260924 20:21:43.768353 30388 ts_manager.cc:194] Registered new tserver with Master: 69330b0b9b1b47afb2451a68ce90b3fd (127.18.172.67:44651)
I20260924 20:21:43.769269 30388 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:54577
I20260924 20:21:43.778430 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:21:43.790747 30607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.780972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45072 (local address 127.18.172.126:34059)
0924 20:21:43.781098 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.781102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.781683 (+ 581us) server_negotiation.cc:407] Connection header received
0924 20:21:43.781885 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.781889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.781937 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.781988 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:21:43.783027 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.783932 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.785246 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.785486 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.785547 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.787289 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:21:43.787293 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:21:43.787296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:21:43.787322 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:21:43.788983 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.789440 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.789445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.789448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.789533 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:21:43.789828 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:21:43.789833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:21:43.789835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:21:43.790022 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:21:43.790168 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.790435 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.790553 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:21:43.794107 30388 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:21:43.797633 30388 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:21:43.817081 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.809169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34441 (local address 127.0.0.1:40860)
0924 20:21:43.809910 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.809960 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.810086 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.811773 (+ 1687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.811779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.811806 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:21:43.811952 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.811960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.813429 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.813433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.815033 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.815044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.816509 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.816513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.816629 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.816669 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.816799 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.816919 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":92,"threads_started":1}
I20260924 20:21:43.817939 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.809976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40860 (local address 127.18.172.65:34441)
0924 20:21:43.811302 (+ 1326us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.811333 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.811360 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:21:43.811447 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.811451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.811617 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.811793 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:21:43.812094 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.813303 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.815173 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.816158 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.816281 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.816442 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.817680 (+ 1238us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.817797 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":81,"threads_started":1}
I20260924 20:21:43.821520 30538 tablet_service.cc:1467] Processing CreateTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 (DEFAULT_TABLE table=test-table [id=a75ca900a03e4a73958c79bc93ca6dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:21:43.822738 30538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaf45c6a5f04c6ab9f4f5731553ed49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:21:43.825098 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.809412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58332 (local address 127.18.172.66:40647)
0924 20:21:43.810514 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.810531 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.812337 (+ 1806us) server_negotiation.cc:407] Connection header received
0924 20:21:43.813234 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.813239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.813441 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.813566 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:21:43.813986 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.815212 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.822072 (+ 6860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.822940 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.823059 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.823207 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.824818 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.824943 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":86,"threads_started":1}
I20260924 20:21:43.825546 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.809258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40647 (local address 127.0.0.1:58332)
0924 20:21:43.812225 (+ 2967us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.812240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.813085 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.813733 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.813736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.813753 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:21:43.813856 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.813862 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.815349 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.815353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.817613 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.817624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.823311 (+ 5687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.823315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.823418 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.823458 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.825301 (+ 1843us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.825386 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2906,"thread_start_us":100,"threads_started":1}
I20260924 20:21:43.828788 30657 tablet_service.cc:1467] Processing CreateTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 (DEFAULT_TABLE table=test-table [id=a75ca900a03e4a73958c79bc93ca6dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:21:43.830816 30657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaf45c6a5f04c6ab9f4f5731553ed49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:21:43.833477 30896 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Bootstrap starting.
I20260924 20:21:43.837333 30896 tablet_bootstrap.cc:654] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Neither blocks nor log segments found. Creating new log.
I20260924 20:21:43.838392 30896 log.cc:824] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:21:43.839171 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.810764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:44651 (local address 127.0.0.1:35756)
0924 20:21:43.817468 (+ 6704us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.817482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.817552 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.818369 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.818372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.818389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:21:43.818460 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.818466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.820677 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.820680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.821884 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.821895 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.838467 (+ 16572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.838474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.838594 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.838623 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.838709 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.838978 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6651,"thread_start_us":86,"threads_started":1}
I20260924 20:21:43.841286 30896 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: No bootstrap required, opened a new log
I20260924 20:21:43.841492 30896 ts_tablet_manager.cc:1397] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 20:21:43.843279 30895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.811017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.18.172.67:44651)
0924 20:21:43.812086 (+ 1069us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.812098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.817734 (+ 5636us) server_negotiation.cc:407] Connection header received
0924 20:21:43.817831 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.817837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.818060 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.818214 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:21:43.818588 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.820542 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.823603 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.838070 (+ 14467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.838215 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.838383 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.842956 (+ 4573us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.843113 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":92,"threads_started":1}
I20260924 20:21:43.846527 30896 raft_consensus.cc:348] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.846976 30896 raft_consensus.cc:374] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:21:43.846531 30808 tablet_service.cc:1467] Processing CreateTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 (DEFAULT_TABLE table=test-table [id=a75ca900a03e4a73958c79bc93ca6dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:21:43.847141 30896 raft_consensus.cc:729] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b562ef9b0ac44a18b1048ef9bced021e, State: Initialized, Role: FOLLOWER
I20260924 20:21:43.847718 30896 consensus_queue.cc:260] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.848531 30808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaf45c6a5f04c6ab9f4f5731553ed49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:21:43.848981 30896 ts_tablet_manager.cc:1428] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260924 20:21:43.849419 30605 heartbeater.cc:503] Master 127.18.172.126:34059 was elected leader, sending a full tablet report...
I20260924 20:21:43.857362 30900 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Bootstrap starting.
I20260924 20:21:43.857474 30898 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Bootstrap starting.
I20260924 20:21:43.860404 30898 tablet_bootstrap.cc:654] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Neither blocks nor log segments found. Creating new log.
I20260924 20:21:43.860486 30900 tablet_bootstrap.cc:654] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Neither blocks nor log segments found. Creating new log.
I20260924 20:21:43.861459 30900 log.cc:824] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:21:43.861459 30898 log.cc:824] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:21:43.863497 30900 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: No bootstrap required, opened a new log
I20260924 20:21:43.863720 30900 ts_tablet_manager.cc:1397] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:21:43.863716 30898 tablet_bootstrap.cc:492] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: No bootstrap required, opened a new log
I20260924 20:21:43.863934 30898 ts_tablet_manager.cc:1397] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260924 20:21:43.867197 30900 raft_consensus.cc:348] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.867499 30900 raft_consensus.cc:374] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:21:43.867566 30900 raft_consensus.cc:729] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69330b0b9b1b47afb2451a68ce90b3fd, State: Initialized, Role: FOLLOWER
I20260924 20:21:43.867897 30898 raft_consensus.cc:348] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.868156 30898 raft_consensus.cc:374] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:21:43.868101 30900 consensus_queue.cc:260] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.868854 30878 heartbeater.cc:503] Master 127.18.172.126:34059 was elected leader, sending a full tablet report...
I20260924 20:21:43.869239 30898 raft_consensus.cc:729] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f0b849c02724bef82fa7348afc35b6e, State: Initialized, Role: FOLLOWER
I20260924 20:21:43.869251 30900 ts_tablet_manager.cc:1428] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:21:43.869751 30898 consensus_queue.cc:260] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.870678 30743 heartbeater.cc:503] Master 127.18.172.126:34059 was elected leader, sending a full tablet report...
I20260924 20:21:43.871556 30898 ts_tablet_manager.cc:1428] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260924 20:21:43.883911 30905 raft_consensus.cc:482] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Sep 24 20:21:43 dist-test-slave-m1qq krb5kdc[29909](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281303, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:21:43.885941 30905 raft_consensus.cc:504] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.887619 30905 leader_election.cc:288] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b562ef9b0ac44a18b1048ef9bced021e (127.18.172.65:34441), 69330b0b9b1b47afb2451a68ce90b3fd (127.18.172.67:44651)
I20260924 20:21:43.893030 30895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.888494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:59169 (local address 127.18.172.67:44651)
0924 20:21:43.888614 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.888621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.888826 (+ 205us) server_negotiation.cc:407] Connection header received
0924 20:21:43.889015 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.889019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.889068 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.889113 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:21:43.889464 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.890142 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.891554 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.892151 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.892224 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.892283 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.892777 (+ 494us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.892859 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:21:43.893719 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.887980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48679 (local address 127.18.172.65:34441)
0924 20:21:43.888434 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.888439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.888452 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:21:43.888503 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.888506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.888559 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.888631 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:21:43.889036 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.889823 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.891414 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.892060 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.892163 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.892225 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.893535 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.893610 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332}
I20260924 20:21:43.893740 30906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.887770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34441 (local address 127.18.172.66:48679)
0924 20:21:43.888147 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.888163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.888259 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.888755 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.888759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.888784 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:21:43.888905 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.888914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.889953 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.889956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.891247 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.891257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.892355 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.892368 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.892518 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.892577 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.893517 (+ 940us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.893596 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":8,"thread_start_us":91,"threads_started":1}
I20260924 20:21:43.894095 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.888112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:44651 (local address 127.18.172.66:59169)
0924 20:21:43.888754 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:43.888766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:21:43.888892 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:21:43.889231 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:21:43.889234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:21:43.889249 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:21:43.889349 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.889354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.890292 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.890295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.891438 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:21:43.891446 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.892381 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.892384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:21:43.892518 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.892673 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 20:21:43.893840 (+ 1167us) client_negotiation.cc:240] Negotiation successful
0924 20:21:43.893934 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":138,"thread_start_us":100,"threads_started":1}
I20260924 20:21:43.894462 30544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaf45c6a5f04c6ab9f4f5731553ed49" candidate_uuid: "7f0b849c02724bef82fa7348afc35b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b562ef9b0ac44a18b1048ef9bced021e" is_pre_election: true
I20260924 20:21:43.894786 30544 raft_consensus.cc:2454] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0b849c02724bef82fa7348afc35b6e in term 0.
I20260924 20:21:43.895152 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaf45c6a5f04c6ab9f4f5731553ed49" candidate_uuid: "7f0b849c02724bef82fa7348afc35b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" is_pre_election: true
I20260924 20:21:43.895308 30628 leader_election.cc:302] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f0b849c02724bef82fa7348afc35b6e, b562ef9b0ac44a18b1048ef9bced021e; no voters:
I20260924 20:21:43.895438 30827 raft_consensus.cc:2454] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f0b849c02724bef82fa7348afc35b6e in term 0.
I20260924 20:21:43.895614 30905 raft_consensus.cc:2781] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:21:43.895694 30905 raft_consensus.cc:482] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:21:43.895741 30905 raft_consensus.cc:3037] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:21:43.897070 30905 raft_consensus.cc:504] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.897420 30905 leader_election.cc:288] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [CANDIDATE]: Term 1 election: Requested vote from peers b562ef9b0ac44a18b1048ef9bced021e (127.18.172.65:34441), 69330b0b9b1b47afb2451a68ce90b3fd (127.18.172.67:44651)
I20260924 20:21:43.897945 30827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaf45c6a5f04c6ab9f4f5731553ed49" candidate_uuid: "7f0b849c02724bef82fa7348afc35b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69330b0b9b1b47afb2451a68ce90b3fd"
I20260924 20:21:43.898064 30827 raft_consensus.cc:3037] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:21:43.899461 30827 raft_consensus.cc:2454] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0b849c02724bef82fa7348afc35b6e in term 1.
I20260924 20:21:43.899891 30544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaf45c6a5f04c6ab9f4f5731553ed49" candidate_uuid: "7f0b849c02724bef82fa7348afc35b6e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b562ef9b0ac44a18b1048ef9bced021e"
I20260924 20:21:43.900018 30544 raft_consensus.cc:3037] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:21:43.901298 30544 raft_consensus.cc:2454] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f0b849c02724bef82fa7348afc35b6e in term 1.
I20260924 20:21:43.901726 30628 leader_election.cc:302] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f0b849c02724bef82fa7348afc35b6e, b562ef9b0ac44a18b1048ef9bced021e; no voters:
I20260924 20:21:43.901943 30905 raft_consensus.cc:2781] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:21:43.902184 30905 raft_consensus.cc:686] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 1 LEADER]: Becoming Leader. State: Replica: 7f0b849c02724bef82fa7348afc35b6e, State: Running, Role: LEADER
I20260924 20:21:43.902575 30905 consensus_queue.cc:237] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } }
I20260924 20:21:43.907109 30389 catalog_manager.cc:5495] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f0b849c02724bef82fa7348afc35b6e (127.18.172.66). New cstate: current_term: 1 leader_uuid: "7f0b849c02724bef82fa7348afc35b6e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f0b849c02724bef82fa7348afc35b6e" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 } health_report { overall_health: UNKNOWN } } }
I20260924 20:21:43.941529 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:43.936754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58344 (local address 127.18.172.66:40647)
0924 20:21:43.936903 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 20:21:43.936907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:21:43.936919 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:21:43.937019 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:21:43.937022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:21:43.937090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:21:43.937192 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:21:43.937597 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.938351 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.939701 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:21:43.940035 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:21:43.940130 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:21:43.940605 (+ 475us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:21:43.941132 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:21:43.941272 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:21:43.941312 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 20:21:43.941367 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 20:21:43.948728 30827 raft_consensus.cc:1261] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 1 FOLLOWER]: Refusing update from remote peer 7f0b849c02724bef82fa7348afc35b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:21:43.948755 30544 raft_consensus.cc:1261] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e [term 1 FOLLOWER]: Refusing update from remote peer 7f0b849c02724bef82fa7348afc35b6e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:21:43.949472 30908 consensus_queue.cc:1035] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "69330b0b9b1b47afb2451a68ce90b3fd" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 44651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:21:43.949656 30905 consensus_queue.cc:1035] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b562ef9b0ac44a18b1048ef9bced021e" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:21:43.956429 30606 tablet.cc:2369] T bdaf45c6a5f04c6ab9f4f5731553ed49 P b562ef9b0ac44a18b1048ef9bced021e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:21:43.963491 30914 mvcc.cc:205] Tried to move back new op lower bound from 7332992220965978112 to 7332992220789596160. Current Snapshot: MvccSnapshot[applied={T|T < 7332992220965978112}]
I20260924 20:21:43.969444 30913 mvcc.cc:205] Tried to move back new op lower bound from 7332992220965978112 to 7332992220789596160. Current Snapshot: MvccSnapshot[applied={T|T < 7332992220965978112}]
I20260924 20:21:43.978952 30389 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:21:43.979175 30389 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45072:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:21:43.981963 30389 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34550353426d4bbd9b94b7d5a077b4a3: Sending DeleteTablet for 3 replicas of tablet bdaf45c6a5f04c6ab9f4f5731553ed49
I20260924 20:21:43.982730 30657 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:21:43 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:58332
I20260924 20:21:43.982811 30538 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:21:43 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:40860
I20260924 20:21:43.982827 30808 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaf45c6a5f04c6ab9f4f5731553ed49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:21:43 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35756
I20260924 20:21:43.983878 30923 tablet_replica.cc:330] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: stopping tablet replica
I20260924 20:21:43.984138 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 30471
I20260924 20:21:43.985042 30923 raft_consensus.cc:2229] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:21:43.985488 30923 raft_consensus.cc:2258] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:21:43.986011 30922 tablet_replica.cc:330] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: stopping tablet replica
I20260924 20:21:43.986302 30922 raft_consensus.cc:2229] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:21:43.986675 30922 raft_consensus.cc:2258] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:21:43.988418 30923 ts_tablet_manager.cc:1905] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:21:43.990547 30922 ts_tablet_manager.cc:1905] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:21:43.991459 30923 ts_tablet_manager.cc:1918] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:21:43.993749 30923 log.cc:1196] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/bdaf45c6a5f04c6ab9f4f5731553ed49
I20260924 20:21:43.994299 30923 ts_tablet_manager.cc:1939] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 69330b0b9b1b47afb2451a68ce90b3fd: Deleting consensus metadata
I20260924 20:21:43.996258 30922 ts_tablet_manager.cc:1918] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:21:43.996369 30922 log.cc:1196] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/bdaf45c6a5f04c6ab9f4f5731553ed49
I20260924 20:21:43.996696 30922 ts_tablet_manager.cc:1939] T bdaf45c6a5f04c6ab9f4f5731553ed49 P 7f0b849c02724bef82fa7348afc35b6e: Deleting consensus metadata
I20260924 20:21:43.998132 30375 catalog_manager.cc:4841] TS 7f0b849c02724bef82fa7348afc35b6e (127.18.172.66:40647): tablet bdaf45c6a5f04c6ab9f4f5731553ed49 (table test-table [id=a75ca900a03e4a73958c79bc93ca6dfd]) successfully deleted
I20260924 20:21:43.998580 30373 catalog_manager.cc:4841] TS 69330b0b9b1b47afb2451a68ce90b3fd (127.18.172.67:44651): tablet bdaf45c6a5f04c6ab9f4f5731553ed49 (table test-table [id=a75ca900a03e4a73958c79bc93ca6dfd]) successfully deleted
W20260924 20:21:43.999351 30373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:34441 (error 108)
I20260924 20:21:44.000442 30891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:21:44.000107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34441 (local address 127.0.0.1:40868)
0924 20:21:44.000237 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 20:21:44.000323 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:21:44.000533 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 30611
W20260924 20:21:44.000699 30373 catalog_manager.cc:4584] TS b562ef9b0ac44a18b1048ef9bced021e (127.18.172.65:34441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bdaf45c6a5f04c6ab9f4f5731553ed49: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34441: connect: Connection refused (error 111)
I20260924 20:21:44.007699 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 30748
I20260924 20:21:44.015580 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 30357
I20260924 20:21:44.026567 19121 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 20:21:44.361467 19121 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 20:21:44.361904 19121 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:21:44.706593 19121 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:21:44.706 UTC [29940] LOG: received smart shutdown request
2026-09-24 20:21:44.710 UTC [29940] LOG: background worker "logical replication launcher" (PID 29949) exited with exit code 1
2026-09-24 20:21:44.710 UTC [29944] LOG: shutting down
2026-09-24 20:21:44.892 UTC [29940] LOG: database system is shut down
2026-09-24T20:21:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48034 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:21:45 dist-test-slave-m1qq krb5kdc[30934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:21:45 dist-test-slave-m1qq krb5kdc[30934](info): set up 2 sockets
Sep 24 20:21:45 dist-test-slave-m1qq krb5kdc[30934](info): commencing operation
krb5kdc: starting...
W20260924 20:21:46.092226 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:21:46 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:21:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangeradmin_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangeradmin_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangerlookup_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangerlookup_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
I20260924 20:21:46.270352 19121 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQK3utR/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/postgres -l logfile start
2026-09-24 20:21:48.997 UTC [30964] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 20:21:48.997 UTC [30964] LOG: listening on IPv4 address "127.18.172.84", port 52309
2026-09-24 20:21:49.001 UTC [30964] LOG: listening on Unix socket "/tmp/.s.PGSQL.52309"
2026-09-24 20:21:49.023 UTC [30967] LOG: database system was shut down at 2026-09-24 20:21:47 UTC
2026-09-24 20:21:49.027 UTC [30964] LOG: database system is ready to accept connections
I20260924 20:21:49.980182 19121 mini_postgres.cc:96] Postgres bound to 52309
2026-09-24 20:21:49.987 UTC [30975] FATAL: database "slave" does not exist
127.18.172.84:52309 - accepting connections
I20260924 20:21:49.987897 19121 mini_ranger.cc:161] Starting Ranger...
I20260924 20:21:50.004532 19121 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 20:21:50.542311 19121 mini_ranger.cc:87] Created ranger Postgres database
I20260924 20:21:50.542390 19121 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin
2026-09-24 20:21:50,801 [I] DB FLAVOR :POSTGRES
2026-09-24 20:21:50,802 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:21:50,802 [I] Checking connection..
2026-09-24 20:21:50,802 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 20:21:51,066 [I] Checking connection passed.
2026-09-24 20:21:51,067 [I] --------- Verifying version history table ---------
2026-09-24 20:21:51,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:51,310 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:21:51,310 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:51,551 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 20:21:51,552 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 20:21:51,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 20:21:51.837 UTC [31040] WARNING: there is no transaction in progress
2026-09-24 20:21:51,843 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 20:21:51,843 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 20:21:52,108 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 20:21:52,109 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:21:52,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 20:21:52,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 20:21:52,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-m1qq.c.gcp-upstream.internal','N') ;"
2026-09-24 20:21:52,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:21:53,119 [I] Table x_portal_user does not exist in database ranger
2026-09-24 20:21:53,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:21:53,384 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 20:21:53,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:21:53,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:21:53,878 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 20:21:53,879 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 20:21:54.920 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:54.938 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:54.957 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:54.973 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:54.990 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.012 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.079 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.086 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.102 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.114 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.126 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.141 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.151 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.159 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.167 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55.175 UTC [31149] WARNING: there is no transaction in progress
2026-09-24 20:21:55,905 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 20:21:55,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 20:21:56,159 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 20:21:56,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 20:21:56,408 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 20:21:56,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 20:21:56,656 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 20:21:56,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 20:21:56,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 20:21:57,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-m1qq.c.gcp-upstream.internal';"
2026-09-24 20:21:57,425 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 20:21:57,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 20:21:57,679 [I] DB_PATCHES have already been applied
I20260924 20:21:57.684826 19121 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQK3utR/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 20:21:57.684895 19121 mini_ranger.cc:193] Using host: 127.18.172.84
I20260924 20:21:57.688439 19121 mini_ranger.cc:227] Ranger admin URL: http://127.18.172.84:47587
Sep 24, 2026 8:21:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:21:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 8:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 8:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 8:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 8:21:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 20:21:58.846791 19121 mini_ranger.cc:160] Time spent starting Ranger: real 8.859s user 0.007s sys 0.003s
Sep 24, 2026 8:22:18 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 8:22:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 8:22:18 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 8:22:18 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 8:22:19 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 20:22:19.749758 19121 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.18.172.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.18.172.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangerkms_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.18.172.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/rangerkms_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.18.172.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.18.172.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.18.172.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/HTTP_127.18.172.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 20:22:19 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 20:22:19.856895 19121 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 20:22:19.867364 19121 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 20:22:20.842909 19121 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 20:22:20.843034 19121 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms
I20260924 20:22:20.843057 19121 mini_ranger_kms.cc:227] Using postgres at 127.18.172.84:52309
2026-09-24 20:22:21,048 [I] DB FLAVOR :POSTGRES
2026-09-24 20:22:21,048 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 20:22:21,048 [I] Checking connection
2026-09-24 20:22:21,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 20:22:21,304 [I] connection success
2026-09-24 20:22:21,305 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 20:22:21,305 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 20:22:21,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 20:22:21,633 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 20:22:21,633 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 20:22:21,633 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 20:22:21,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.18.172.84:52309/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 20:22:21,912 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 20:22:22.397024 19121 mini_ranger_kms.cc:310] Created kms service
I20260924 20:22:22.589301 19121 mini_ranger_kms.cc:326] Created kudu user
I20260924 20:22:22.633800 19121 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 20:22:22 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281339, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:23.045818 19121 mini_ranger_kms.cc:384] Added ranger policy
I20260924 20:22:23.046072 19121 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQK3utR/build/debug/bin/postgresql.jar:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQK3utR/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 20:22:23.046098 19121 mini_ranger_kms.cc:240] Using host: 127.18.172.84
I20260924 20:22:23.050758 19121 mini_ranger_kms.cc:276] Ranger KMS PID: 31329
I20260924 20:22:23.050848 19121 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.18.172.84:42455
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 8:22:24 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 20:22:25.304926 19121 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.448s user 0.002s sys 0.187s
I20260924 20:22:25.305074 19121 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 20:22:25.305161 19121 mini_ranger_kms.cc:401] 127.18.172.84:42455/kms/v1/keys
Sep 24 20:22:25 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 20:22:25 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
Sep 24 20:22:26 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281345, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.18.172.84@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
Sep 24 20:22:28 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:22:28.398910 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41487
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41487
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:28.509814 31380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:28.510078 31380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:28.510119 31380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:28.514201 31380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:28.514264 31380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:28.514283 31380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:28.514299 31380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:28.514353 31380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:28.519805 31380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
--ranger_kms_url=127.18.172.84:42455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41487
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:28.520848 31380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:28.521837 31380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:22:28.531037 31380 server_base.cc:1034] running on GCE node
W20260924 20:22:28.530963 31388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:28.530978 31385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:28.530961 31386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:28.531886 31380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:28.532960 31380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:28.534168 31380 hybrid_clock.cc:648] HybridClock initialized: now 1790281348534137 us; error 47 us; skew 500 ppm
Sep 24 20:22:28 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281348, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:28.536940 31380 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:28.537729 31380 webserver.cc:460] Webserver started at http://127.18.172.126:33283/ using document root <none> and password file <none>
I20260924 20:22:28.538203 31380 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:28.538251 31380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:28.538405 31380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:22:28 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281348, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:28.598507 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "0cad6144a4ab487b82afa39189d1202d"
format_stamp: "Formatted at 2026-09-24 20:22:28 on dist-test-slave-m1qq"
server_key: "577fe4aed0da16c77380814c769602e1"
server_key_iv: "f440c187ca1f766719dde43e6bf0eaf4"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:28.599156 31380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "0cad6144a4ab487b82afa39189d1202d"
format_stamp: "Formatted at 2026-09-24 20:22:28 on dist-test-slave-m1qq"
server_key: "577fe4aed0da16c77380814c769602e1"
server_key_iv: "f440c187ca1f766719dde43e6bf0eaf4"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:28.603137 31380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:22:28.622071 31395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:28.623647 31380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:28.623818 31380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "0cad6144a4ab487b82afa39189d1202d"
format_stamp: "Formatted at 2026-09-24 20:22:28 on dist-test-slave-m1qq"
server_key: "577fe4aed0da16c77380814c769602e1"
server_key_iv: "f440c187ca1f766719dde43e6bf0eaf4"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:28.623958 31380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:28.645130 31380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:28.648350 31380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:28.648566 31380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:28.656474 31380 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41487
I20260924 20:22:28.656474 31447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41487 every 8 connection(s)
I20260924 20:22:28.657444 31380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:28.660238 31448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:28.665616 31448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Bootstrap starting.
I20260924 20:22:28.666706 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31380
I20260924 20:22:28.666837 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:22:28.667191 19121 external_mini_cluster.cc:1428] Setting key 7d55ce84faf03ced59aaab665cbc28cb
I20260924 20:22:28.668349 31448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:28.669335 31448 log.cc:824] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:28.671581 31448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: No bootstrap required, opened a new log
I20260924 20:22:28.675977 31448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:28.676263 31448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:28.676331 31448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad6144a4ab487b82afa39189d1202d, State: Initialized, Role: FOLLOWER
Sep 24 20:22:28 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:28.676873 31448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:28.677059 31448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:28.677146 31448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:28.677244 31448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:28.678407 31448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:28.678745 31448 leader_election.cc:302] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cad6144a4ab487b82afa39189d1202d; no voters:
I20260924 20:22:28.679123 31448 leader_election.cc:288] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:22:28.679206 31453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:28.679420 31453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 LEADER]: Becoming Leader. State: Replica: 0cad6144a4ab487b82afa39189d1202d, State: Running, Role: LEADER
I20260924 20:22:28.679790 31453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:28.680461 31448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:28.681367 31454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cad6144a4ab487b82afa39189d1202d. Latest consensus state: current_term: 1 leader_uuid: "0cad6144a4ab487b82afa39189d1202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } } }
I20260924 20:22:28.681511 31454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:28.681713 31454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0cad6144a4ab487b82afa39189d1202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } } }
I20260924 20:22:28.681779 31454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:28.682762 31451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:28.668374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39914 (local address 127.18.172.126:41487)
0924 20:22:28.668853 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:28.668873 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:28.668916 (+ 43us) server_negotiation.cc:407] Connection header received
0924 20:22:28.669755 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:28.669786 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:28.670138 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:28.670506 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:28.671246 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:28.672685 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:28.673528 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:28.674215 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:28.674377 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:28.677094 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:28.677101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:28.677117 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:28.677152 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:28.679032 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:28.679564 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:28.679570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:28.679576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:28.679701 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:28.680097 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:28.680102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:28.680104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:28.680491 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:28.680718 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:28.681461 (+ 743us) server_negotiation.cc:299] Negotiation successful
0924 20:22:28.681734 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":139,"threads_started":1}
I20260924 20:22:28.682972 31462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:28.683143 31448 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:22:28.688710 31462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:28.695587 31462 catalog_manager.cc:1347] Generated new cluster ID: a40904a45232459f9987a4c0001a2ce2
I20260924 20:22:28.695655 31462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:28.723670 31462 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:22:28.725169 31462 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:28.732506 31462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Generated new TSK 0
I20260924 20:22:28.733410 31462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:22:29.395584 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:29.541923 31493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:29.542203 31493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:29.542294 31493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:29.546873 31493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:29.546959 31493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:29.547078 31493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:29.552742 31493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:29.553858 31493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:29.554905 31493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:29.566265 31498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:29.566219 31499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:29.566274 31501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:29.566571 31493 server_base.cc:1034] running on GCE node
I20260924 20:22:29.567257 31493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:29.568059 31493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:29.569281 31493 hybrid_clock.cc:648] HybridClock initialized: now 1790281349569200 us; error 89 us; skew 500 ppm
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:29.572741 31493 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:29.573709 31493 webserver.cc:460] Webserver started at http://127.18.172.65:36501/ using document root <none> and password file <none>
I20260924 20:22:29.574281 31493 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:29.574379 31493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:29.574617 31493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:29.589531 31493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "3cd5a2bc6d8143409b105c027fe44b10"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "99a307b58fa8d5979a280788b2a541a0"
server_key_iv: "3b58ea69ffa77686d62109f3ae9cb74c"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.590289 31493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "3cd5a2bc6d8143409b105c027fe44b10"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "99a307b58fa8d5979a280788b2a541a0"
server_key_iv: "3b58ea69ffa77686d62109f3ae9cb74c"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.596935 31493 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 20:22:29.613391 31510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:29.615229 31493 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:22:29.615367 31493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "3cd5a2bc6d8143409b105c027fe44b10"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "99a307b58fa8d5979a280788b2a541a0"
server_key_iv: "3b58ea69ffa77686d62109f3ae9cb74c"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.615505 31493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:29.634452 31493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:29.638005 31493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:29.638221 31493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:29.638913 31493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:29.640013 31493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:29.640072 31493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:29.640125 31493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:29.640182 31493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 20:22:29.651099 31624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:41747 every 8 connection(s)
I20260924 20:22:29.651582 31493 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:41747
I20260924 20:22:29.652714 31493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:29.654356 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31493
I20260924 20:22:29.654476 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:22:29.654775 19121 external_mini_cluster.cc:1428] Setting key b3892d9fa582ffbdb0022da2988f6b8a
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:22:29.687152 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:29.659305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:50635 (local address 127.18.172.126:41487)
0924 20:22:29.669991 (+ 10686us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:29.670000 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:29.670019 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:22:29.670087 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:29.670090 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:29.670146 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:29.671919 (+ 1773us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:29.672042 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:29.672899 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:29.674667 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:29.675061 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:29.675858 (+ 797us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:29.678198 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:29.678202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:29.678206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:29.678234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:29.683578 (+ 5344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:29.684403 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:29.684407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:29.684408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:29.684528 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:29.685009 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:29.685012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:29.685014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:29.685281 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:29.685375 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:29.686731 (+ 1356us) server_negotiation.cc:299] Negotiation successful
0924 20:22:29.686849 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10591,"thread_start_us":97,"threads_started":1}
I20260924 20:22:29.687835 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:29.655290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.65:50635)
0924 20:22:29.655834 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:29.655892 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:29.656948 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:29.670558 (+ 13610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:29.670568 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:29.670955 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:29.671727 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:29.671754 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:29.673067 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:29.673071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:29.674504 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:29.674516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:29.675247 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:29.675251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:29.675497 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:29.676184 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:29.676205 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:29.677973 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:29.683776 (+ 5803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:29.683784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:29.683807 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:29.684239 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:29.684649 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:29.684652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:29.684654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:29.684879 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:29.685427 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:29.685433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:29.685833 (+ 400us) client_negotiation.cc:769] Sending connection context
0924 20:22:29.686097 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 20:22:29.686507 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:22:29.689513 31625 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:29.689914 31625 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:29.690589 31625 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:29.692411 31412 ts_manager.cc:194] Registered new tserver with Master: 3cd5a2bc6d8143409b105c027fe44b10 (127.18.172.65:41747)
I20260924 20:22:29.693949 31412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:50635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:22:29.707098 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 20:22:29.869793 31633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:29.870214 31633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:29.870354 31633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:29.876645 31633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:29.876719 31633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:29.876806 31633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:22:29.885577 31633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:29.887164 31633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:29.888626 31633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:29.901294 31641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:29.901398 31643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:29.901294 31640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:29.901660 31633 server_base.cc:1034] running on GCE node
I20260924 20:22:29.902304 31633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:29.903126 31633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:29.904752 31633 hybrid_clock.cc:648] HybridClock initialized: now 1790281349904717 us; error 47 us; skew 500 ppm
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:29.912590 31633 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:22:29.913865 31633 webserver.cc:460] Webserver started at http://127.18.172.66:32929/ using document root <none> and password file <none>
I20260924 20:22:29.914642 31633 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:29.914740 31633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:29.915120 31633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:22:29 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:29.931879 31633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "ee7c708c307042e2aa0f8a428b7895cb"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "d86383a92500a79a3a0ed4540d43f54b"
server_key_iv: "70f026e3ad70ae3670e6037992d87530"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.932816 31633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "ee7c708c307042e2aa0f8a428b7895cb"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "d86383a92500a79a3a0ed4540d43f54b"
server_key_iv: "70f026e3ad70ae3670e6037992d87530"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.938161 31633 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 20:22:29.964449 31650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:29.971271 31633 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.003s sys 0.000s
I20260924 20:22:29.971474 31633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "ee7c708c307042e2aa0f8a428b7895cb"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "d86383a92500a79a3a0ed4540d43f54b"
server_key_iv: "70f026e3ad70ae3670e6037992d87530"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:29.971632 31633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:29.992094 31633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:29.995632 31633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:29.995862 31633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:29.996539 31633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:29.997503 31633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:29.997578 31633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:29.997680 31633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:29.997732 31633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.009737 31633 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:38071
I20260924 20:22:30.009761 31763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:38071 every 8 connection(s)
I20260924 20:22:30.010859 31633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:22:30.020113 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31633
I20260924 20:22:30.020257 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:22:30.020696 19121 external_mini_cluster.cc:1428] Setting key f249a9830f2a8db01024fe7e2769df61
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:30.026940 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.013016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:54423 (local address 127.18.172.126:41487)
0924 20:22:30.013478 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:30.013483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:30.014159 (+ 676us) server_negotiation.cc:407] Connection header received
0924 20:22:30.015311 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:30.015315 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:30.015374 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:30.015460 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:30.016631 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.017229 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.018478 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.018751 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.018870 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.022009 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:30.022013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:30.022016 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:30.022043 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:30.023996 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.024771 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.024777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.024781 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.024883 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.025353 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.025358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.025362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.025625 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:30.025712 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:30.026489 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 20:22:30.026665 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260924 20:22:30.029057 31766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.013484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.66:54423)
0924 20:22:30.013991 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:30.014048 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:30.015078 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:30.015649 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:30.015656 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:30.015947 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:30.016469 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.016484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.017366 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.017371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.018332 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.018342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.018922 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.018926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.019183 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.019897 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:30.019922 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:30.021833 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:30.024143 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.024150 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.024167 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.024620 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.025025 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.025029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.025032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.025232 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.025804 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:30.025814 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:30.026183 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 20:22:30.027436 (+ 1253us) client_negotiation.cc:240] Negotiation successful
0924 20:22:30.027831 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":149,"threads_started":1}
I20260924 20:22:30.030710 31764 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:30.031072 31764 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:30.031767 31764 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:30.033017 31412 ts_manager.cc:194] Registered new tserver with Master: ee7c708c307042e2aa0f8a428b7895cb (127.18.172.66:38071)
I20260924 20:22:30.033653 31412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:54423
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:22:30.058823 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:30.205596 31769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:30.205884 31769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:30.205961 31769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:30.210716 31769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:30.210819 31769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:30.210928 31769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:22:30.216877 31769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:30.218278 31769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:30.219429 31769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:30.229835 31777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:30.230147 31775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:30.230374 31769 server_base.cc:1034] running on GCE node
W20260924 20:22:30.230445 31774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:30.230794 31769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:30.231539 31769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:30.232733 31769 hybrid_clock.cc:648] HybridClock initialized: now 1790281350232690 us; error 49 us; skew 500 ppm
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:30.236644 31769 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:22:30.238108 31769 webserver.cc:460] Webserver started at http://127.18.172.67:37987/ using document root <none> and password file <none>
I20260924 20:22:30.238889 31769 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:30.239041 31769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:30.239280 31769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:30.257110 31769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "60dcddf2ece74ac58add6a7de6b477b7"
format_stamp: "Formatted at 2026-09-24 20:22:30 on dist-test-slave-m1qq"
server_key: "fdee5eedb264a6b00fdf0e68e9e0d2b3"
server_key_iv: "e067c7958fe15565f13661f388d4adc0"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:30.258049 31769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "60dcddf2ece74ac58add6a7de6b477b7"
format_stamp: "Formatted at 2026-09-24 20:22:30 on dist-test-slave-m1qq"
server_key: "fdee5eedb264a6b00fdf0e68e9e0d2b3"
server_key_iv: "e067c7958fe15565f13661f388d4adc0"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:30.268649 31769 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.000s
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281349, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:30.297765 31784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.300546 31769 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260924 20:22:30.300700 31769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "60dcddf2ece74ac58add6a7de6b477b7"
format_stamp: "Formatted at 2026-09-24 20:22:30 on dist-test-slave-m1qq"
server_key: "fdee5eedb264a6b00fdf0e68e9e0d2b3"
server_key_iv: "e067c7958fe15565f13661f388d4adc0"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:30.300863 31769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:30.332643 31769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:30.336535 31769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:30.336740 31769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:30.337469 31769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:30.338660 31769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:30.338724 31769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.338790 31769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:30.338824 31769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.349150 31769 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42799
I20260924 20:22:30.350306 31769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:22:30.357714 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31769
I20260924 20:22:30.357864 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:22:30.358196 19121 external_mini_cluster.cc:1428] Setting key d7c474c7984e8c9a25f52442c3caf899
I20260924 20:22:30.359716 31897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42799 every 8 connection(s)
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:30.379383 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.358736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:46017 (local address 127.18.172.126:41487)
0924 20:22:30.358873 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:30.358878 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:30.363564 (+ 4686us) server_negotiation.cc:407] Connection header received
0924 20:22:30.364853 (+ 1289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:30.364859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:30.364914 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:30.365003 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:30.366873 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.367774 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.369556 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.369917 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.369996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.374914 (+ 4918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:30.374918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:30.374921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:30.374951 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:30.376786 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.377500 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.377504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.377506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.377590 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.377997 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.378002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.378007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.378235 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:30.378331 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:30.379097 (+ 766us) server_negotiation.cc:299] Negotiation successful
0924 20:22:30.379218 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:22:30.380970 31900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.357584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.67:46017)
0924 20:22:30.363360 (+ 5776us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:30.363438 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:30.364581 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:30.365341 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:30.365352 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:30.365750 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:30.366689 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.366715 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.367949 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.367952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.369397 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.369408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.370208 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.370211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.370431 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.372896 (+ 2465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:30.372920 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:30.374714 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:30.376924 (+ 2210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.376931 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.376951 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.377386 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.377708 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.377711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.377713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.377891 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.378459 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:30.378466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:30.378795 (+ 329us) client_negotiation.cc:769] Sending connection context
0924 20:22:30.379600 (+ 805us) client_negotiation.cc:240] Negotiation successful
0924 20:22:30.379939 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5482,"thread_start_us":178,"threads_started":1}
I20260924 20:22:30.383200 31898 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:30.383457 31898 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:30.384030 31898 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:30.386394 31412 ts_manager.cc:194] Registered new tserver with Master: 60dcddf2ece74ac58add6a7de6b477b7 (127.18.172.67:42799)
I20260924 20:22:30.387135 31412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:46017
I20260924 20:22:30.397936 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:22:30.398886 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31493
I20260924 20:22:30.404767 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31633
I20260924 20:22:30.412005 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31769
I20260924 20:22:30.418366 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31380
I20260924 20:22:30.424934 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41487
--webserver_interface=127.18.172.126
--webserver_port=33283
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41487
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:30.538187 31901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:30.538568 31901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:30.538643 31901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:30.542644 31901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:30.542701 31901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:30.542719 31901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:30.542735 31901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:30.542750 31901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:30.548300 31901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41487
--ranger_config_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=33283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:30.549363 31901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:30.550376 31901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:30.558959 31906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:30.558933 31907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:30.559096 31901 server_base.cc:1034] running on GCE node
W20260924 20:22:30.559257 31909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:30.559773 31901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:30.560923 31901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:30.562115 31901 hybrid_clock.cc:648] HybridClock initialized: now 1790281350562091 us; error 37 us; skew 500 ppm
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:30.564986 31901 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:30.565881 31901 webserver.cc:460] Webserver started at http://127.18.172.126:33283/ using document root <none> and password file <none>
I20260924 20:22:30.566404 31901 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:30.566496 31901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:30.613983 31901 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:22:30.616477 31917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.617777 31901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:22:30.617923 31901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "0cad6144a4ab487b82afa39189d1202d"
format_stamp: "Formatted at 2026-09-24 20:22:28 on dist-test-slave-m1qq"
server_key: "577fe4aed0da16c77380814c769602e1"
server_key_iv: "f440c187ca1f766719dde43e6bf0eaf4"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:30.618501 31901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:30.649657 31901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:30.652743 31901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:30.652948 31901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:30.659888 31901 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41487
I20260924 20:22:30.659927 31969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41487 every 8 connection(s)
I20260924 20:22:30.660970 31901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:30.662676 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31901
I20260924 20:22:30.663213 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:41747
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:41487
--webserver_port=36501
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:30.665892 31970 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:22:30.667626 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Bootstrap starting.
I20260924 20:22:30.683898 31970 log.cc:824] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:30.690366 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:22:30.690779 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Bootstrap complete.
I20260924 20:22:30.694893 31970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:30.695340 31970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cad6144a4ab487b82afa39189d1202d, State: Initialized, Role: FOLLOWER
I20260924 20:22:30.695839 31970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:30.695991 31970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:30.696092 31970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:30.696220 31970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:22:30.698153 31970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:30.698477 31970 leader_election.cc:302] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0cad6144a4ab487b82afa39189d1202d; no voters:
I20260924 20:22:30.698772 31970 leader_election.cc:288] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:22:30.698910 31974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:22:30.699193 31974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [term 2 LEADER]: Becoming Leader. State: Replica: 0cad6144a4ab487b82afa39189d1202d, State: Running, Role: LEADER
I20260924 20:22:30.699488 31974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } }
I20260924 20:22:30.700037 31970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:30.700842 31975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0cad6144a4ab487b82afa39189d1202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } } }
I20260924 20:22:30.700974 31975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:30.701344 31976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cad6144a4ab487b82afa39189d1202d. Latest consensus state: current_term: 2 leader_uuid: "0cad6144a4ab487b82afa39189d1202d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0cad6144a4ab487b82afa39189d1202d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41487 } } }
I20260924 20:22:30.701490 31976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:30.701798 31982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:30.705013 31982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:30.705499 31970 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 20:22:30.705706 31982 catalog_manager.cc:1259] Loaded cluster ID: a40904a45232459f9987a4c0001a2ce2
I20260924 20:22:30.705757 31982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:30.710870 31982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:30.712061 31982 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Loaded TSK: 0
I20260924 20:22:30.713335 31982 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:22:30.788954 31972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:30.789307 31972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:30.789386 31972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:30.794979 31972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:30.795112 31972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:30.795213 31972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:30.801644 31972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:41747
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=36501
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:30.802845 31972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:30.803918 31972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:30.814296 32008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:30.814608 32010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:30.814291 32007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:30.816542 31972 server_base.cc:1034] running on GCE node
I20260924 20:22:30.817051 31972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:30.817730 31972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:30.819079 31972 hybrid_clock.cc:648] HybridClock initialized: now 1790281350819026 us; error 63 us; skew 500 ppm
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:30.822474 31972 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:30.823529 31972 webserver.cc:460] Webserver started at http://127.18.172.65:36501/ using document root <none> and password file <none>
I20260924 20:22:30.824117 31972 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:30.824222 31972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:30.852921 31972 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260924 20:22:30.855985 32018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.857311 31972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:22:30.857442 31972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "3cd5a2bc6d8143409b105c027fe44b10"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "99a307b58fa8d5979a280788b2a541a0"
server_key_iv: "3b58ea69ffa77686d62109f3ae9cb74c"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:30.858067 31972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:30.874042 31972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:30.877804 31972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:30.878012 31972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:30.878726 31972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:30.880949 31972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:30.881031 31972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:30.881174 31972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:30.881207 31972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 20:22:30.896358 31972 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:41747
I20260924 20:22:30.897374 31972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:30.899384 32131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:41747 every 8 connection(s)
I20260924 20:22:30.900854 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 31972
I20260924 20:22:30.901345 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:38071
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:41487
--webserver_port=32929
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:22:30 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281350, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:30.924639 32134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.899822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.65:52615)
0924 20:22:30.900304 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:30.900357 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:30.904666 (+ 4309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:30.910810 (+ 6144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:30.910820 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:30.911210 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:30.911677 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.911691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.913050 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.913056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.914511 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:30.914523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.915587 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.915592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:30.915796 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.916497 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:30.916518 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:30.918273 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:30.920797 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.920803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.920818 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.921170 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.921589 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:30.921592 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:30.921595 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:30.921765 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:30.922677 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:30.922683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:30.923027 (+ 344us) client_negotiation.cc:769] Sending connection context
0924 20:22:30.923243 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 20:22:30.923574 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":149,"threads_started":1}
I20260924 20:22:30.927168 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:30.903408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:52615 (local address 127.18.172.126:41487)
0924 20:22:30.909735 (+ 6327us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:30.909756 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:30.909796 (+ 40us) server_negotiation.cc:407] Connection header received
0924 20:22:30.910038 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:30.910051 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:30.910369 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:30.910731 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:30.911850 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.912915 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.914659 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:30.915397 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:30.916228 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:30.918496 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:30.918502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:30.918519 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:30.918552 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:30.920658 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.921323 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.921329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.921334 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.921474 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:30.921900 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:30.921903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:30.921905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:30.922438 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:30.925758 (+ 3320us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:30.925918 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 20:22:30.926228 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6133,"thread_start_us":110,"threads_started":1}
I20260924 20:22:30.930161 32132 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:30.930413 32132 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:30.931026 32132 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
W20260924 20:22:30.933203 32132 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:41487 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:22:30.933323 32132 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:30.933502 32132 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:30.934312 32132 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:30.934482 32132 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
W20260924 20:22:30.935285 32132 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:22:31.060976 32135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:31.061371 32135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:31.061456 32135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:31.067569 32135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:31.067646 32135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:31.067735 32135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:22:31.074858 32135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:38071
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=32929
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:31.076045 32135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:31.077282 32135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:31.088016 32142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:31.088205 32141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:31.088047 32144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:31.088817 32135 server_base.cc:1034] running on GCE node
I20260924 20:22:31.089243 32135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:31.089913 32135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:31.091090 32135 hybrid_clock.cc:648] HybridClock initialized: now 1790281351091048 us; error 44 us; skew 500 ppm
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:31.093902 32135 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:22:31.094843 32135 webserver.cc:460] Webserver started at http://127.18.172.66:32929/ using document root <none> and password file <none>
I20260924 20:22:31.095485 32135 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:31.095594 32135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:31.112536 32135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:31.115195 32152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.116590 32135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:31.116690 32135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "ee7c708c307042e2aa0f8a428b7895cb"
format_stamp: "Formatted at 2026-09-24 20:22:29 on dist-test-slave-m1qq"
server_key: "d86383a92500a79a3a0ed4540d43f54b"
server_key_iv: "70f026e3ad70ae3670e6037992d87530"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:31.117311 32135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:31.150873 32135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:31.154053 32135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:31.154265 32135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:31.154990 32135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:31.156173 32135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:31.156240 32135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.156307 32135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:31.156358 32135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.168731 32135 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:38071
I20260924 20:22:31.170042 32135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:22:31.172875 32265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:38071 every 8 connection(s)
I20260924 20:22:31.176493 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 32135
I20260924 20:22:31.176985 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.18.172.84:42455/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:42799
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:41487
--webserver_port=37987
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:31.189693 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:31.174177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:46743 (local address 127.18.172.126:41487)
0924 20:22:31.174310 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:31.174314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:31.175284 (+ 970us) server_negotiation.cc:407] Connection header received
0924 20:22:31.176367 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:31.176372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:31.176432 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:31.176494 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:31.178376 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.179290 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.180715 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.181128 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.181245 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:31.184512 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:31.184518 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:31.184520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:31.184558 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:31.186622 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.187413 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.187419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.187422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.187523 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.188092 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.188097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.188100 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.188382 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:31.188481 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:31.189304 (+ 823us) server_negotiation.cc:299] Negotiation successful
0924 20:22:31.189465 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:22:31.191919 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:31.174636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.66:46743)
0924 20:22:31.175126 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:31.175179 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:31.176083 (+ 904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:31.176954 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:31.176963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:31.177459 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:31.178178 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:31.178208 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.179479 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.179482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:31.180584 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:31.180594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.181433 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.181440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:31.181670 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:31.182496 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:31.182520 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:31.184274 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:31.186777 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:31.186786 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:31.186809 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:31.187275 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:31.187658 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:31.187664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:31.187666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:31.187948 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:31.188561 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:31.188570 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:31.188959 (+ 389us) client_negotiation.cc:769] Sending connection context
0924 20:22:31.190178 (+ 1219us) client_negotiation.cc:240] Negotiation successful
0924 20:22:31.190641 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":143,"threads_started":1}
I20260924 20:22:31.193482 32266 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:31.193742 32266 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:31.194234 32266 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
W20260924 20:22:31.195797 32266 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:41487 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 20:22:31.195930 32266 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:31.196113 32266 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:31.196909 32266 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:31.197105 32266 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
W20260924 20:22:31.197826 32266 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:22:31.324231 32269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:31.324529 32269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:31.324621 32269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:31.328998 32269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:31.329057 32269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:31.329141 32269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:22:31.334842 32269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.18.172.84:42455/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:42799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=37987
--tserver_master_addrs=127.18.172.126:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:31.336161 32269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:31.337222 32269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:31.350560 32276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:31.350813 32278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:31.351387 32275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:31.352276 32269 server_base.cc:1034] running on GCE node
I20260924 20:22:31.352867 32269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:31.353516 32269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:31.356848 32269 hybrid_clock.cc:648] HybridClock initialized: now 1790281351356808 us; error 49 us; skew 500 ppm
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:31.359720 32269 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:22:31.360743 32269 webserver.cc:460] Webserver started at http://127.18.172.67:37987/ using document root <none> and password file <none>
I20260924 20:22:31.361457 32269 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:31.361574 32269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:31.387193 32269 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:31.390522 32286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.392134 32269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:22:31.392284 32269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "60dcddf2ece74ac58add6a7de6b477b7"
format_stamp: "Formatted at 2026-09-24 20:22:30 on dist-test-slave-m1qq"
server_key: "fdee5eedb264a6b00fdf0e68e9e0d2b3"
server_key_iv: "e067c7958fe15565f13661f388d4adc0"
server_key_version: "kuduclusterkey@0"
I20260924 20:22:31.392854 32269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:31.421150 32269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:31.425315 32269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:31.425560 32269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:31.426223 32269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:31.427309 32269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:31.427376 32269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.427429 32269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:31.427459 32269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:31.480131 32269 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42799
I20260924 20:22:31.481549 32269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:22:31.482478 32399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42799 every 8 connection(s)
I20260924 20:22:31.488566 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 32269
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:31.498560 32136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:31.484373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:38393 (local address 127.18.172.126:41487)
0924 20:22:31.484536 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:31.484542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:31.485506 (+ 964us) server_negotiation.cc:407] Connection header received
0924 20:22:31.486780 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:31.486786 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:31.486851 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:31.486910 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:31.488805 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.489606 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.491293 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.491630 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.491707 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:31.494491 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:31.494495 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:31.494498 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:31.494522 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:31.496118 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.496819 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.496824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.496827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.496943 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.497392 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.497395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.497397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.497601 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:31.497697 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:31.498272 (+ 575us) server_negotiation.cc:299] Negotiation successful
0924 20:22:31.498391 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:22:31.499853 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:31.484840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41487 (local address 127.18.172.67:38393)
0924 20:22:31.485309 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:31.485364 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:31.486310 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:31.487335 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:31.487343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:31.487868 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:31.488610 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:31.488632 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.490183 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.490186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:31.491170 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:31.491182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.491924 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.491930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:31.492144 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:31.492809 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:31.492829 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:31.494289 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:31.496236 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:31.496242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:31.496260 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:31.496679 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:31.497079 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:31.497084 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:31.497089 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:31.497275 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:31.497720 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:31.497727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:31.498022 (+ 295us) client_negotiation.cc:769] Sending connection context
0924 20:22:31.498216 (+ 194us) client_negotiation.cc:240] Negotiation successful
0924 20:22:31.498537 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":141,"threads_started":1}
I20260924 20:22:31.501190 32400 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41487
I20260924 20:22:31.501441 32400 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:31.501926 32400 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:31.503422 31934 ts_manager.cc:194] Registered new tserver with Master: 60dcddf2ece74ac58add6a7de6b477b7 (127.18.172.67:42799)
I20260924 20:22:31.504717 32411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:31.489858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39918 (local address 127.18.172.126:41487)
0924 20:22:31.490125 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:31.490134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:31.490149 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:22:31.491290 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:31.491293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:31.491341 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:31.491413 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:31.492390 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.493112 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.494946 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:31.495396 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:31.495530 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:31.498756 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:31.498759 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:31.498762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:31.498783 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:31.502886 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.503359 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.503364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.503366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.503450 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:31.503804 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:31.503807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:31.503809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:31.504023 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:31.504201 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:31.504429 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 20:22:31.504558 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":125,"threads_started":1}
I20260924 20:22:31.504987 31934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:38393
Sep 24 20:22:31 dist-test-slave-m1qq krb5kdc[30934](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:31.935534 32132 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:31.935850 32132 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:31.936772 31932 ts_manager.cc:194] Registered new tserver with Master: 3cd5a2bc6d8143409b105c027fe44b10 (127.18.172.65:41747)
I20260924 20:22:31.937578 31932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:52615
I20260924 20:22:32.198002 32266 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:32.198284 32266 heartbeater.cc:511] Master 127.18.172.126:41487 requested a full tablet report, sending...
I20260924 20:22:32.199265 31932 ts_manager.cc:194] Registered new tserver with Master: ee7c708c307042e2aa0f8a428b7895cb (127.18.172.66:38071)
I20260924 20:22:32.200083 31932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:46743
I20260924 20:22:32.207732 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:22:32.219267 32422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.209330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.18.172.126:41487)
0924 20:22:32.209604 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.209609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.209626 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:22:32.209848 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.209853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.209919 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.210035 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:32.210984 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.211821 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.213227 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.213569 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.213696 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.215371 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:32.215375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:32.215378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:32.215404 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:32.217245 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:32.217775 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:32.217779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:32.217781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:32.217865 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:32.218183 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:32.218187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:32.218189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:32.218400 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:32.218522 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.218889 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.219068 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
Sep 24 20:22:32 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:32.223928 31932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:22:32.229002 31932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:22:32.258774 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.244156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.18.172.67:42799)
0924 20:22:32.250759 (+ 6603us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.250764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.250805 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:22:32.251139 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.251146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.251558 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.251904 (+ 346us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:32.252298 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.254526 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.255927 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.257315 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.257605 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.257751 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.258373 (+ 622us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.258478 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6507,"thread_start_us":81,"threads_started":1}
I20260924 20:22:32.262001 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.247310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.18.172.66:38071)
0924 20:22:32.248419 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.248426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.250441 (+ 2015us) server_negotiation.cc:407] Connection header received
0924 20:22:32.251696 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.251702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.251917 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.252043 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:32.252397 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.253468 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.259076 (+ 5608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.260431 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.261217 (+ 786us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.261383 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.261449 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.261848 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":74,"threads_started":1}
I20260924 20:22:32.265405 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.246841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.18.172.65:41747)
0924 20:22:32.248184 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.248191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.248218 (+ 27us) server_negotiation.cc:407] Connection header received
0924 20:22:32.248674 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.248683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.248925 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.249164 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:32.249661 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.250823 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.252532 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.253180 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.253325 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.253494 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.265120 (+ 11626us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.265247 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1236,"thread_start_us":89,"threads_started":1}
I20260924 20:22:32.265789 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.244040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42799 (local address 127.0.0.1:59794)
0924 20:22:32.247657 (+ 3617us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.248565 (+ 908us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:32.248645 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:32.252061 (+ 3416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:32.252064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:32.252081 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:32.252173 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.252179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.254655 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.254658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.255796 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.255805 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.257939 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.257945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.258039 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.258065 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:22:32.258134 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 20:22:32.265651 (+ 7517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3421,"mutex_wait_us":7413,"thread_start_us":118,"threads_started":1}
I20260924 20:22:32.267481 32427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.246303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:38071 (local address 127.0.0.1:56302)
0924 20:22:32.250323 (+ 4020us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.250335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:32.250456 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:32.252173 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:32.252175 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:32.252189 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:32.252284 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.252290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.254080 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.254083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.255275 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.255286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.260950 (+ 5664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.260954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.261067 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.261092 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:22:32.261153 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 20:22:32.267317 (+ 6164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3962,"mutex_wait_us":6053,"thread_start_us":86,"threads_started":1}
I20260924 20:22:32.268162 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.245927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:41747 (local address 127.0.0.1:56156)
0924 20:22:32.247751 (+ 1824us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.247818 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:32.248447 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:32.249346 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:32.249353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:32.249377 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:32.249522 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.249529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.250956 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.250961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.252374 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.252388 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.253313 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.253317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.253691 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.253763 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 20:22:32.253890 (+ 127us) client_negotiation.cc:240] Negotiation successful
0924 20:22:32.268027 (+ 14137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1772,"thread_start_us":104,"threads_started":1}
I20260924 20:22:32.269227 32328 tablet_service.cc:1467] Processing CreateTablet for tablet 816e68b37e5b4abd9142db25f808ff0c (DEFAULT_TABLE table=test-table [id=1e02f641daf84bd7983c7e0ebb40cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:32.270428 32328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e68b37e5b4abd9142db25f808ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:32.277875 32200 tablet_service.cc:1467] Processing CreateTablet for tablet 816e68b37e5b4abd9142db25f808ff0c (DEFAULT_TABLE table=test-table [id=1e02f641daf84bd7983c7e0ebb40cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:32.279106 32200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e68b37e5b4abd9142db25f808ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:32.285617 32066 tablet_service.cc:1467] Processing CreateTablet for tablet 816e68b37e5b4abd9142db25f808ff0c (DEFAULT_TABLE table=test-table [id=1e02f641daf84bd7983c7e0ebb40cd49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:32.286793 32066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 816e68b37e5b4abd9142db25f808ff0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:32.304098 32433 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Bootstrap starting.
I20260924 20:22:32.306523 32433 tablet_bootstrap.cc:654] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:32.308231 32433 log.cc:824] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:32.309312 32434 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Bootstrap starting.
I20260924 20:22:32.309650 32435 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Bootstrap starting.
I20260924 20:22:32.311750 32434 tablet_bootstrap.cc:654] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:32.312016 32435 tablet_bootstrap.cc:654] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:32.312829 32434 log.cc:824] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:32.312963 32435 log.cc:824] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:32.315948 32434 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: No bootstrap required, opened a new log
I20260924 20:22:32.315950 32433 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: No bootstrap required, opened a new log
I20260924 20:22:32.316172 32434 ts_tablet_manager.cc:1397] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:22:32.316172 32433 ts_tablet_manager.cc:1397] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260924 20:22:32.320658 32434 raft_consensus.cc:348] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.320724 32433 raft_consensus.cc:348] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.321044 32434 raft_consensus.cc:374] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:32.321091 32434 raft_consensus.cc:729] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee7c708c307042e2aa0f8a428b7895cb, State: Initialized, Role: FOLLOWER
I20260924 20:22:32.321087 32433 raft_consensus.cc:374] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:32.321151 32433 raft_consensus.cc:729] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd5a2bc6d8143409b105c027fe44b10, State: Initialized, Role: FOLLOWER
I20260924 20:22:32.321656 32433 consensus_queue.cc:260] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.321656 32434 consensus_queue.cc:260] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.322929 32433 ts_tablet_manager.cc:1428] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:22:32.322966 32434 ts_tablet_manager.cc:1428] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260924 20:22:32.323360 32132 heartbeater.cc:503] Master 127.18.172.126:41487 was elected leader, sending a full tablet report...
I20260924 20:22:32.324455 32266 heartbeater.cc:503] Master 127.18.172.126:41487 was elected leader, sending a full tablet report...
I20260924 20:22:32.329376 32435 tablet_bootstrap.cc:492] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: No bootstrap required, opened a new log
I20260924 20:22:32.329696 32435 ts_tablet_manager.cc:1397] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20260924 20:22:32.334692 32435 raft_consensus.cc:348] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.335122 32435 raft_consensus.cc:374] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:32.335220 32435 raft_consensus.cc:729] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60dcddf2ece74ac58add6a7de6b477b7, State: Initialized, Role: FOLLOWER
I20260924 20:22:32.335793 32435 consensus_queue.cc:260] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.336489 32400 heartbeater.cc:503] Master 127.18.172.126:41487 was elected leader, sending a full tablet report...
I20260924 20:22:32.337059 32435 ts_tablet_manager.cc:1428] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260924 20:22:32.399714 32133 tablet.cc:2369] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:22:32.424278 32267 tablet.cc:2369] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:22:32.484769 32401 tablet.cc:2369] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 20:22:32 dist-test-slave-m1qq krb5kdc[30934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281351, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for HTTP/127.18.172.84@KRBTEST.COM
I20260924 20:22:32.645334 32440 raft_consensus.cc:482] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:32.645596 32440 raft_consensus.cc:504] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.647055 32440 leader_election.cc:288] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60dcddf2ece74ac58add6a7de6b477b7 (127.18.172.67:42799), 3cd5a2bc6d8143409b105c027fe44b10 (127.18.172.65:41747)
I20260924 20:22:32.653482 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.648031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:43043 (local address 127.18.172.65:41747)
0924 20:22:32.648157 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.648164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.649014 (+ 850us) server_negotiation.cc:407] Connection header received
0924 20:22:32.649227 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.649230 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.649280 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.649329 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:32.649798 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.650578 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.652167 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.652770 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.652889 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.652941 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.653262 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.653338 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 20:22:32.654098 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.647521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:41747 (local address 127.18.172.66:43043)
0924 20:22:32.648922 (+ 1401us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.648937 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:32.649098 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:32.649501 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:32.649504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:32.649522 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:32.649669 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.649676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.650721 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.650724 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.652021 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.652033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.653006 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.653010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.653111 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.653137 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:22:32.653874 (+ 737us) client_negotiation.cc:240] Negotiation successful
0924 20:22:32.653956 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"thread_start_us":131,"threads_started":1}
I20260924 20:22:32.654942 32084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e68b37e5b4abd9142db25f808ff0c" candidate_uuid: "ee7c708c307042e2aa0f8a428b7895cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd5a2bc6d8143409b105c027fe44b10" is_pre_election: true
I20260924 20:22:32.655294 32084 raft_consensus.cc:2454] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee7c708c307042e2aa0f8a428b7895cb in term 0.
I20260924 20:22:32.655908 32156 leader_election.cc:302] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd5a2bc6d8143409b105c027fe44b10, ee7c708c307042e2aa0f8a428b7895cb; no voters:
I20260924 20:22:32.656209 32440 raft_consensus.cc:2781] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:22:32.656281 32440 raft_consensus.cc:482] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:22:32.656337 32440 raft_consensus.cc:3037] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:32.657620 32440 raft_consensus.cc:504] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.657961 32440 leader_election.cc:288] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [CANDIDATE]: Term 1 election: Requested vote from peers 60dcddf2ece74ac58add6a7de6b477b7 (127.18.172.67:42799), 3cd5a2bc6d8143409b105c027fe44b10 (127.18.172.65:41747)
I20260924 20:22:32.658499 32084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e68b37e5b4abd9142db25f808ff0c" candidate_uuid: "ee7c708c307042e2aa0f8a428b7895cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd5a2bc6d8143409b105c027fe44b10"
I20260924 20:22:32.658617 32084 raft_consensus.cc:3037] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:32.658731 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.648644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:43035 (local address 127.18.172.67:42799)
0924 20:22:32.648757 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.648761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.652946 (+ 4185us) server_negotiation.cc:407] Connection header received
0924 20:22:32.653763 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.653766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.653821 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.653870 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:32.655137 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.655879 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.657368 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.657958 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.658051 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.658100 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.658537 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.658615 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:22:32.659404 32445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.648279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42799 (local address 127.18.172.66:43035)
0924 20:22:32.652843 (+ 4564us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.652857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:32.653015 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:32.653999 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:32.654001 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:32.654018 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:32.654142 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.654147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.656023 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.656026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.657208 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:32.657220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.658278 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.658282 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:32.658394 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.658418 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:22:32.658915 (+ 497us) client_negotiation.cc:240] Negotiation successful
0924 20:22:32.658989 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4500,"thread_start_us":103,"threads_started":1}
I20260924 20:22:32.659978 32084 raft_consensus.cc:2454] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7c708c307042e2aa0f8a428b7895cb in term 1.
I20260924 20:22:32.660315 32344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e68b37e5b4abd9142db25f808ff0c" candidate_uuid: "ee7c708c307042e2aa0f8a428b7895cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dcddf2ece74ac58add6a7de6b477b7"
I20260924 20:22:32.660545 32156 leader_election.cc:302] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd5a2bc6d8143409b105c027fe44b10, ee7c708c307042e2aa0f8a428b7895cb; no voters:
I20260924 20:22:32.660549 32344 raft_consensus.cc:3037] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:32.660705 32440 raft_consensus.cc:2781] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:32.660902 32440 raft_consensus.cc:686] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 1 LEADER]: Becoming Leader. State: Replica: ee7c708c307042e2aa0f8a428b7895cb, State: Running, Role: LEADER
I20260924 20:22:32.661254 32440 consensus_queue.cc:237] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } }
I20260924 20:22:32.661785 32344 raft_consensus.cc:2454] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee7c708c307042e2aa0f8a428b7895cb in term 1.
I20260924 20:22:32.660323 32342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "816e68b37e5b4abd9142db25f808ff0c" candidate_uuid: "ee7c708c307042e2aa0f8a428b7895cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60dcddf2ece74ac58add6a7de6b477b7" is_pre_election: true
I20260924 20:22:32.662122 32342 raft_consensus.cc:2362] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ee7c708c307042e2aa0f8a428b7895cb in term 1. Re-sending same reply.
I20260924 20:22:32.664535 31934 catalog_manager.cc:5495] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb reported cstate change: term changed from 0 to 1, leader changed from <none> to ee7c708c307042e2aa0f8a428b7895cb (127.18.172.66). New cstate: current_term: 1 leader_uuid: "ee7c708c307042e2aa0f8a428b7895cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee7c708c307042e2aa0f8a428b7895cb" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38071 } health_report { overall_health: HEALTHY } } }
I20260924 20:22:32.706050 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.700917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56312 (local address 127.18.172.66:38071)
0924 20:22:32.701069 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:32.701075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:32.701096 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:22:32.701228 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:32.701231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:32.701335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:32.701489 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:22:32.701788 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.702632 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.703978 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:32.704458 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:32.704588 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:32.704970 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:22:32.705541 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:22:32.705677 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:32.705806 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 20:22:32.705871 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:22:32.713668 32342 raft_consensus.cc:1261] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 1 FOLLOWER]: Refusing update from remote peer ee7c708c307042e2aa0f8a428b7895cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:32.713726 32084 raft_consensus.cc:1261] T 816e68b37e5b4abd9142db25f808ff0c P 3cd5a2bc6d8143409b105c027fe44b10 [term 1 FOLLOWER]: Refusing update from remote peer ee7c708c307042e2aa0f8a428b7895cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:32.714485 32446 consensus_queue.cc:1035] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd5a2bc6d8143409b105c027fe44b10" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 41747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:32.714809 32440 consensus_queue.cc:1035] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "60dcddf2ece74ac58add6a7de6b477b7" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:32.723104 32449 mvcc.cc:205] Tried to move back new op lower bound from 7332992420705234944 to 7332992420504047616. Current Snapshot: MvccSnapshot[applied={T|T < 7332992420705234944}]
I20260924 20:22:32.726090 32451 mvcc.cc:205] Tried to move back new op lower bound from 7332992420705234944 to 7332992420504047616. Current Snapshot: MvccSnapshot[applied={T|T < 7332992420705234944}]
I20260924 20:22:32.732291 32450 mvcc.cc:205] Tried to move back new op lower bound from 7332992420705234944 to 7332992420504047616. Current Snapshot: MvccSnapshot[applied={T|T < 7332992420705234944}]
I20260924 20:22:32.737345 31934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:32.737541 31934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:32.740307 31934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cad6144a4ab487b82afa39189d1202d: Sending DeleteTablet for 3 replicas of tablet 816e68b37e5b4abd9142db25f808ff0c
I20260924 20:22:32.741603 32328 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e68b37e5b4abd9142db25f808ff0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:32 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:59794
I20260924 20:22:32.742945 32066 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e68b37e5b4abd9142db25f808ff0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:32 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:56156
I20260924 20:22:32.743474 32200 tablet_service.cc:1514] Processing DeleteTablet for tablet 816e68b37e5b4abd9142db25f808ff0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:32 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:56302
I20260924 20:22:32.744597 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31972
I20260924 20:22:32.747213 32460 tablet_replica.cc:330] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: stopping tablet replica
I20260924 20:22:32.747643 32460 raft_consensus.cc:2229] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:32.748083 32460 raft_consensus.cc:2258] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:32.749477 32460 ts_tablet_manager.cc:1905] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:32.751245 32462 tablet_replica.cc:330] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb: stopping tablet replica
I20260924 20:22:32.751961 32460 ts_tablet_manager.cc:1918] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:32.752113 32460 log.cc:1196] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/816e68b37e5b4abd9142db25f808ff0c
I20260924 20:22:32.752465 32460 ts_tablet_manager.cc:1939] T 816e68b37e5b4abd9142db25f808ff0c P 60dcddf2ece74ac58add6a7de6b477b7: Deleting consensus metadata
I20260924 20:22:32.752451 32462 raft_consensus.cc:2229] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:22:32.752897 32462 raft_consensus.cc:2258] T 816e68b37e5b4abd9142db25f808ff0c P ee7c708c307042e2aa0f8a428b7895cb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:22:32.753686 31921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:41747 (error 108)
I20260924 20:22:32.753818 31920 catalog_manager.cc:4841] TS 60dcddf2ece74ac58add6a7de6b477b7 (127.18.172.67:42799): tablet 816e68b37e5b4abd9142db25f808ff0c (table test-table [id=1e02f641daf84bd7983c7e0ebb40cd49]) successfully deleted
I20260924 20:22:32.754971 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 32135
I20260924 20:22:32.755071 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.754737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:41747 (local address 127.0.0.1:56168)
0924 20:22:32.754878 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.754951 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:41747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:22:32.756539 31921 catalog_manager.cc:4584] TS 3cd5a2bc6d8143409b105c027fe44b10 (127.18.172.65:41747): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 816e68b37e5b4abd9142db25f808ff0c: Network error: Client connection negotiation failed: client connection to 127.18.172.65:41747: connect: Connection refused (error 111)
I20260924 20:22:32.762465 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 32269
I20260924 20:22:32.764846 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:32.762703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:38071 (local address 127.0.0.1:56320)
0924 20:22:32.764723 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:32.764773 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:38071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1941}
I20260924 20:22:32.770677 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 31901
I20260924 20:22:32.778820 19121 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 20:22:33.123936 19121 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 20:22:33.124047 19121 mini_ranger.cc:66] Stopping Ranger...
I20260924 20:22:33.469027 19121 mini_ranger.cc:68] Stopped Ranger
2026-09-24 20:22:33.469 UTC [30964] LOG: received smart shutdown request
2026-09-24 20:22:33.473 UTC [30964] LOG: background worker "logical replication launcher" (PID 30973) exited with exit code 1
2026-09-24 20:22:33.473 UTC [30968] LOG: shutting down
2026-09-24 20:22:33.673 UTC [30964] LOG: database system is shut down
2026-09-24T20:22:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48782 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:22:33 dist-test-slave-m1qq krb5kdc[32477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:22:33 dist-test-slave-m1qq krb5kdc[32477](info): set up 2 sockets
Sep 24 20:22:33 dist-test-slave-m1qq krb5kdc[32477](info): commencing operation
krb5kdc: starting...
W20260924 20:22:34.888432 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:22:34 dist-test-slave-m1qq krb5kdc[32477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:22:35.015659 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41545
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:35.124130 32490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:35.124445 32490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:35.124511 32490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:35.128539 32490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:35.128620 32490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:35.128659 32490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:35.128718 32490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:35.128757 32490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:35.133944 32490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:35.135067 32490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:35.136113 32490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:35.144779 32495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.144888 32490 server_base.cc:1034] running on GCE node
W20260924 20:22:35.144824 32498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:35.144831 32496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.145740 32490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:35.146883 32490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:35.148164 32490 hybrid_clock.cc:648] HybridClock initialized: now 1790281355148133 us; error 46 us; skew 500 ppm
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:35.153821 32490 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:35.154863 32490 webserver.cc:460] Webserver started at http://127.18.172.126:41181/ using document root <none> and password file <none>
I20260924 20:22:35.155467 32490 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:35.155558 32490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:35.155791 32490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:35.157836 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "af26a6c076374b38b2e4be807ff30e40"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "d3c5be6738c42c4a8921c2d405c12330"
server_key_iv: "2ee77dec193b4c700ac87afa2497236d"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.158402 32490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "af26a6c076374b38b2e4be807ff30e40"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "d3c5be6738c42c4a8921c2d405c12330"
server_key_iv: "2ee77dec193b4c700ac87afa2497236d"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.162745 32490 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260924 20:22:35.165525 32505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:35.166769 32490 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:22:35.166910 32490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "af26a6c076374b38b2e4be807ff30e40"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "d3c5be6738c42c4a8921c2d405c12330"
server_key_iv: "2ee77dec193b4c700ac87afa2497236d"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.167049 32490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:35.185618 32490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:35.188879 32490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:35.189090 32490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:35.196182 32490 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41545
I20260924 20:22:35.196568 32557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41545 every 8 connection(s)
I20260924 20:22:35.197171 32490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:35.200133 32558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:35.201843 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 32490
I20260924 20:22:35.202008 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:22:35.202272 19121 external_mini_cluster.cc:1428] Setting key f9ef944d12ee0660a30be8fe2feb091a
I20260924 20:22:35.204090 32558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.125@KRBTEST.COM" created.
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:35.220178 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.206495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41545 (local address 127.0.0.1:51856)
0924 20:22:35.206975 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.207072 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.207898 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.208766 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.208770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.209099 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.209355 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.209368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.211143 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.211146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.211882 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.211891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.212541 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.212544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.212639 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.213340 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.213362 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.215149 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.217149 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.217158 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.217170 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.217410 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.217739 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.217743 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.217745 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.217866 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.218599 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.218604 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.218727 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.218875 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.219372 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":181,"thread_start_us":133,"threads_started":1}
I20260924 20:22:35.220183 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.206496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.18.172.126:41545)
0924 20:22:35.207077 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.207091 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.207193 (+ 102us) server_negotiation.cc:407] Connection header received
0924 20:22:35.208234 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.208243 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.208497 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.208743 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.209515 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.210966 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.212007 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.212363 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.212488 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.215302 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.215308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.215318 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.215337 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.217010 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.217523 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.217528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.217530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.217637 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.217973 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.217976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.217978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.218471 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.218624 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.219026 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.219304 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":422,"spinlock_wait_cycles":9984,"thread_start_us":157,"threads_started":1}
I20260924 20:22:35.222805 32558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.223603 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.223309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60406)
0924 20:22:35.223427 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.223502 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 20:22:35.223811 32507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
I20260924 20:22:35.224677 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.224396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60408)
0924 20:22:35.224576 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.224627 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260924 20:22:35.224812 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:39355: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
I20260924 20:22:35.232100 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:39355
--webserver_interface=127.18.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:35.274068 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } attempt: 1
I20260924 20:22:35.275215 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.274811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60414)
0924 20:22:35.275064 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.275137 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 20:22:35.275964 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.275725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60420)
0924 20:22:35.275837 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.275890 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 20:22:35.276094 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:39355: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
W20260924 20:22:35.343721 32565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:35.344007 32565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:35.344089 32565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:22:35.344276 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } attempt: 2
I20260924 20:22:35.345049 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.344758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60424)
0924 20:22:35.344902 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.344975 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 20:22:35.345767 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.345548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60428)
0924 20:22:35.345656 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.345710 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 20:22:35.345899 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:39355: Network error: Client connection negotiation failed: client connection to 127.18.172.125:39355: connect: Connection refused (error 111)
W20260924 20:22:35.348397 32565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:35.348480 32565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:35.348528 32565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:35.348577 32565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:35.348614 32565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:35.354110 32565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:39355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:35.355263 32565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:35.356323 32565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:35.364944 32571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:35.364933 32573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.365011 32565 server_base.cc:1034] running on GCE node
W20260924 20:22:35.364933 32570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.365746 32565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:35.366830 32565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:35.368041 32565 hybrid_clock.cc:648] HybridClock initialized: now 1790281355368029 us; error 32 us; skew 500 ppm
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:35.371177 32565 init.cc:377] Logged in from keytab as kudu/127.18.172.125@KRBTEST.COM (short username kudu)
I20260924 20:22:35.372159 32565 webserver.cc:460] Webserver started at http://127.18.172.125:35127/ using document root <none> and password file <none>
I20260924 20:22:35.372780 32565 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:35.372876 32565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:35.373088 32565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:35.375245 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data/instance:
uuid: "df46b1a419a04474be236274a188d879"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "e329368a6e82e48e14197098de9ada97"
server_key_iv: "e61bd7ced429d66e0982959a7dbfe891"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.375823 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal/instance:
uuid: "df46b1a419a04474be236274a188d879"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "e329368a6e82e48e14197098de9ada97"
server_key_iv: "e61bd7ced429d66e0982959a7dbfe891"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.379490 32565 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:22:35.381871 32580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:35.383220 32565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:22:35.383364 32565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "df46b1a419a04474be236274a188d879"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "e329368a6e82e48e14197098de9ada97"
server_key_iv: "e61bd7ced429d66e0982959a7dbfe891"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.383471 32565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:35.408218 32565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:35.411126 32565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:35.411281 32565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:35.418265 32565 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.125:39355
I20260924 20:22:35.418603 32632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.125:39355 every 8 connection(s)
I20260924 20:22:35.419263 32565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
I20260924 20:22:35.421877 32633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:35.425547 32633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.428635 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 32565
I20260924 20:22:35.428784 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-1/wal/instance
I20260924 20:22:35.429097 19121 external_mini_cluster.cc:1428] Setting key c9031ca044a8cea43e335ab2f4b0f0bd
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.124@KRBTEST.COM; defaulting to no policy
I20260924 20:22:35.437075 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.124@KRBTEST.COM" created.
I20260924 20:22:35.440479 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.427550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.18.172.126:41545)
0924 20:22:35.427718 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.427723 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.428581 (+ 858us) server_negotiation.cc:407] Connection header received
0924 20:22:35.429726 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.429729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.429776 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.429850 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.431360 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.432132 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.433227 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.433464 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.433587 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.435967 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.435971 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.435973 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.435997 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.438003 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.438636 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.438639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.438640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.438713 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.439133 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.439137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.439138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.439339 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.439441 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.440087 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.440209 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:22:35.441241 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.427907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41545 (local address 127.0.0.1:51860)
0924 20:22:35.428328 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.428427 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.429510 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.430046 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.430057 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.430563 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.431212 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.431227 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.432243 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.432246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.433091 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.433100 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.433621 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.433626 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.433792 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.434367 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.434383 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.435787 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.438118 (+ 2331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.438123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.438135 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.438532 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.438811 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.438816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.438818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.438987 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.439440 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.439445 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.439669 (+ 224us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.439998 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.440323 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260924 20:22:35.443568 32633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:22:35.448096 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.437513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60430)
0924 20:22:35.437642 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.437658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.437733 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.438340 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.438343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.438580 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.438705 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.438711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.439916 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.439920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.440769 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.440783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.441650 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.441654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.441761 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.442229 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.442248 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.444512 (+ 2264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.446533 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.446536 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.446538 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.446732 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.447058 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.447063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.447066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.447152 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.447583 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.447586 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.447681 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.447796 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.447931 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 20:22:35.448228 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.437633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.18.172.125:39355)
0924 20:22:35.437930 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.437943 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.437968 (+ 25us) server_negotiation.cc:407] Connection header received
0924 20:22:35.438008 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.438013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.438212 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.438312 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.438813 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.439789 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.440913 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.441520 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.441642 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.444659 (+ 3017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.444669 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.444682 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.444706 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.446409 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.446850 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.446855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.446859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.446936 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.447243 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.447248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.447250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.447477 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.447615 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.447905 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.448046 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":128,"threads_started":1}
I20260924 20:22:35.450181 32558 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.450843 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.450614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50698)
0924 20:22:35.450726 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.450777 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 20:22:35.451679 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.451452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50708)
0924 20:22:35.451591 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.451631 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 20:22:35.451882 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:22:35.455994 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.444092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60446)
0924 20:22:35.444398 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.444416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.444497 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.445292 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.445295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.445515 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.445658 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.445668 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.446839 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.446843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.447728 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.447744 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.448491 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.448497 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.448607 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.449032 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.449049 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.452768 (+ 3719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.454590 (+ 1822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.454596 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.454600 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.454820 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.455179 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.455181 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.455183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.455254 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.455596 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.455598 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.455671 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.455793 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.455885 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260924 20:22:35.456142 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.444112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60446 (local address 127.18.172.125:39355)
0924 20:22:35.444775 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.444784 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.444802 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:22:35.444835 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.444838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.444883 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.444975 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.445796 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.446710 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.447888 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.448286 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.448388 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.452920 (+ 4532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.452923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.452925 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.452946 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.454473 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.454931 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.454933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.454935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.455070 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.455339 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.455342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.455343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.455514 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.455625 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.455862 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.455963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":123,"threads_started":1}
I20260924 20:22:35.456861 32633 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.457691 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.457429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50722)
0924 20:22:35.457547 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.457629 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:22:35.457911 32581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
I20260924 20:22:35.458828 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.458627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50738)
0924 20:22:35.458732 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.458773 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:22:35.458995 32633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
I20260924 20:22:35.459324 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:46719
--webserver_interface=127.18.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:35.483114 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 1
I20260924 20:22:35.484027 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.483730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50744)
0924 20:22:35.483885 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.483950 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 20:22:35.484802 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.484562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50752)
0924 20:22:35.484691 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.484740 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:22:35.484942 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
I20260924 20:22:35.508275 32633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 1
I20260924 20:22:35.509200 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.508893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50760)
0924 20:22:35.509061 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.509124 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 20:22:35.509908 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.509695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50776)
0924 20:22:35.509804 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.509855 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260924 20:22:35.510039 32633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
I20260924 20:22:35.560118 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 2
I20260924 20:22:35.561082 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.560694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50784)
0924 20:22:35.560906 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.560982 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 20:22:35.561904 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.561688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50792)
0924 20:22:35.561798 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.561849 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:22:35.562039 32558 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
W20260924 20:22:35.571946 32641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:35.572237 32641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:35.572340 32641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:35.576383 32641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:35.576464 32641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:35.576498 32641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:35.576557 32641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:35.576594 32641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:35.578204 32633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 2
I20260924 20:22:35.579036 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.578709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50794)
0924 20:22:35.578852 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.578919 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 20:22:35.579720 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.579514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50810)
0924 20:22:35.579617 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.579662 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:22:35.579829 32633 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:46719: Network error: Client connection negotiation failed: client connection to 127.18.172.124:46719: connect: Connection refused (error 111)
I20260924 20:22:35.582134 32641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41545,127.18.172.125:39355,127.18.172.124:46719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:46719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:35.583259 32641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:35.584280 32641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:35.592947 32646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.593017 32641 server_base.cc:1034] running on GCE node
W20260924 20:22:35.592897 32649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:35.592897 32647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:35.593812 32641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:35.594879 32641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:35.596040 32641 hybrid_clock.cc:648] HybridClock initialized: now 1790281355596022 us; error 28 us; skew 500 ppm
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:35.598841 32641 init.cc:377] Logged in from keytab as kudu/127.18.172.124@KRBTEST.COM (short username kudu)
I20260924 20:22:35.599860 32641 webserver.cc:460] Webserver started at http://127.18.172.124:43399/ using document root <none> and password file <none>
I20260924 20:22:35.600414 32641 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:35.600478 32641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:35.600646 32641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:35.602730 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data/instance:
uuid: "44308ce7ada843eab93ecfd9be252f94"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "c6b0ba1090c4705e70a63384f0d9c5d2"
server_key_iv: "9bcb216da58ba3e20e2bd65d338283b6"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.603330 32641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal/instance:
uuid: "44308ce7ada843eab93ecfd9be252f94"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "c6b0ba1090c4705e70a63384f0d9c5d2"
server_key_iv: "9bcb216da58ba3e20e2bd65d338283b6"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.607342 32641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:22:35.609787 32656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:35.611157 32641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:22:35.611285 32641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "44308ce7ada843eab93ecfd9be252f94"
format_stamp: "Formatted at 2026-09-24 20:22:35 on dist-test-slave-m1qq"
server_key: "c6b0ba1090c4705e70a63384f0d9c5d2"
server_key_iv: "9bcb216da58ba3e20e2bd65d338283b6"
server_key_version: "encryptionkey@0"
I20260924 20:22:35.611398 32641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:35.641927 32641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:35.645339 32641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:35.645604 32641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:35.653463 32641 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.124:46719
I20260924 20:22:35.653746 32708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.124:46719 every 8 connection(s)
I20260924 20:22:35.654537 32641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
I20260924 20:22:35.656004 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 32641
I20260924 20:22:35.656178 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790281188136637-19121-0/minicluster-data/master-2/wal/instance
I20260924 20:22:35.656606 19121 external_mini_cluster.cc:1428] Setting key ec9a903abaee5a745a8c19aedaf3eff8
I20260924 20:22:35.657841 32709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:35.661207 32558 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 3
I20260924 20:22:35.662215 32709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:35.670979 32633 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } attempt: 3
I20260924 20:22:35.673005 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.661639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50816)
0924 20:22:35.661822 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.661836 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.661912 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.662257 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.662260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.662481 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.662593 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.662600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.663715 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.663718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.664506 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.664515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.665083 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.665087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.665175 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.665594 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.665605 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.667297 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.670448 (+ 3151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.670452 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.670454 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.670701 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.671144 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.671149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.671155 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.671276 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.672180 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.672184 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.672290 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.672402 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.672821 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 20:22:35.673997 32712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.657935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.18.172.124:46719)
0924 20:22:35.658333 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.658357 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.658466 (+ 109us) server_negotiation.cc:407] Connection header received
0924 20:22:35.659176 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.659194 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.659521 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.659792 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.660673 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.661920 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.663071 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.663769 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.663920 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.667516 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.667522 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.667528 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.667568 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.670343 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.670959 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.670963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.670964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.671127 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.671440 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.671443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.671445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.672045 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.672240 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.672692 (+ 452us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.672984 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":253,"spinlock_wait_cycles":12288,"thread_start_us":125,"threads_started":1}
I20260924 20:22:35.674002 32713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.661692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.18.172.124:46719)
0924 20:22:35.662045 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.662053 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.662072 (+ 19us) server_negotiation.cc:407] Connection header received
0924 20:22:35.662097 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.662099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.662130 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.662229 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.662748 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.663571 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.664636 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.664948 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.665077 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.667494 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.667503 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.667520 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.667599 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.670324 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.670823 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.670831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.670837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.670989 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.671420 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.671424 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.671427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.672045 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.672542 (+ 497us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.672695 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.673079 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":106,"server-negotiator.queue_time_us":281,"spinlock_wait_cycles":14464,"thread_start_us":134,"threads_started":1}
I20260924 20:22:35.675961 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.664816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.18.172.126:41545)
0924 20:22:35.664967 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.664971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.664986 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:22:35.665023 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.665029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.665079 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.665165 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.666103 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.666619 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.667589 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.667949 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.668022 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.670930 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.670973 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.670976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.671022 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.672676 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.673590 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.673593 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.673595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.673679 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.674016 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.674019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.674021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.674303 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.674383 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.675738 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.675855 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:22:35.676594 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.664021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41545 (local address 127.0.0.1:51870)
0924 20:22:35.664366 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.664416 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.664514 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.665337 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.665345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.665806 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.665979 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.665995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.666732 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.666735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.667396 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.667406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.668070 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.668076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.668206 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.668690 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.668710 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.670805 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.673149 (+ 2344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.673156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.673160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.673481 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.673790 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.673793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.673795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.673897 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.675362 (+ 1465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.675371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.675493 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.676165 (+ 672us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.676369 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":93,"threads_started":1}
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
I20260924 20:22:35.677923 32709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.680347 32558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: Bootstrap starting.
I20260924 20:22:35.681838 32635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.671683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50820)
0924 20:22:35.671835 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.671851 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.671935 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.673657 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.673660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.673831 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.673934 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.673940 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.674799 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.674804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.675422 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.675431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.675993 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.675997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.676077 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.676513 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.676529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.678106 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.680210 (+ 2104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.680213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.680215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.680479 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.680800 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.680804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.680816 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.680913 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.681335 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.681338 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.681444 (+ 106us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.681538 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.681669 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 20:22:35.682566 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.672071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50820 (local address 127.18.172.124:46719)
0924 20:22:35.673417 (+ 1346us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.673429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.673441 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:22:35.673471 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.673474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.673511 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.674062 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.674148 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.674685 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.675550 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.675852 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.676284 (+ 432us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.678225 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.678229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.678231 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.678254 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.680082 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.680608 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.680611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.680613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.680693 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.681034 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.681037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.681039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.681228 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.682131 (+ 903us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.682206 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.682318 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":94,"threads_started":1}
I20260924 20:22:35.683457 32558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: Neither blocks nor log segments found. Creating new log.
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:22:35.684460 32558 log.cc:824] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:35.686704 32558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: No bootstrap required, opened a new log
I20260924 20:22:35.686311 32633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: Bootstrap starting.
I20260924 20:22:35.689803 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.678808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.18.172.126:41545)
0924 20:22:35.678919 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.678923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.678936 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:22:35.679037 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.679044 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.679097 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.679177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.679995 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.680639 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.681726 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.682040 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.682109 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.685707 (+ 3598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.685710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.685713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.685735 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.687538 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.688100 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.688107 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.688110 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.688199 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.688485 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.688488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.688490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.688692 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.688854 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.689535 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.689648 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:22:35.690100 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.678553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.18.172.125:39355)
0924 20:22:35.678684 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.678690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.678703 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:22:35.678736 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.678739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.678787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.678881 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.679538 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.680401 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.681560 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.681946 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.682032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.686165 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.686171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.686176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.686212 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.688057 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.688581 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.688586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.688589 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.688678 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.689026 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.689030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.689032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.689264 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.689338 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.689713 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.689838 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:22:35.690747 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.678316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:39355 (local address 127.0.0.1:60452)
0924 20:22:35.678460 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.678476 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.678556 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.679044 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.679049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.679236 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.679379 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.679388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.680548 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.680555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.681430 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.681440 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.682181 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.682185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.682298 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.683319 (+ 1021us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.683337 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.685994 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.688191 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.688194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.688196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.688446 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.688808 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.688811 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.688813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.688914 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.689418 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.689421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.689517 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.690287 (+ 770us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.690532 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 20:22:35.690748 32633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:35.691468 32709 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:22:35.691393 32558 raft_consensus.cc:348] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.691641 32558 raft_consensus.cc:374] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:35.691704 32558 raft_consensus.cc:729] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af26a6c076374b38b2e4be807ff30e40, State: Initialized, Role: FOLLOWER
I20260924 20:22:35.691815 32633 log.cc:824] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:35.692240 32558 consensus_queue.cc:260] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.693596 32558 sys_catalog.cc:564] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:35.694321 32633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: No bootstrap required, opened a new log
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281355, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:22:35 dist-test-slave-m1qq krb5kdc[32477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
W20260924 20:22:35.698843 32730 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:22:35.698915 32730 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:22:35.698736 32490 master_runner.cc:186] Error getting master registration for 127.18.172.125:39355: OK, instance_id { permanent_uuid: "df46b1a419a04474be236274a188d879" instance_seqno: 1790281355416190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:22:35.702642 32490 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:22:35.702723 32714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.691913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:46719 (local address 127.0.0.1:50828)
0924 20:22:35.692325 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:35.692339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:35.692423 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:35.692638 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:35.692642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:35.692808 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:35.692932 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.692939 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.693925 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.693928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.694660 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:35.694670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.695243 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.695246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:35.695315 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.696239 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:35.696253 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:35.699122 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:35.701342 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.701357 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.701359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.701583 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.701846 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:35.701849 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:35.701851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:35.701934 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:35.702289 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:35.702292 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:35.702364 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 20:22:35.702435 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 20:22:35.702552 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":103,"threads_started":1}
I20260924 20:22:35.702836 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.691121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60462 (local address 127.18.172.125:39355)
0924 20:22:35.691218 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.691224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.691594 (+ 370us) server_negotiation.cc:407] Connection header received
0924 20:22:35.691796 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.691800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.691846 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.692780 (+ 934us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.692869 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.693555 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.694741 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.695106 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.695220 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.698532 (+ 3312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.698536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.698539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.698563 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.700257 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.700899 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.700935 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.700940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.701061 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.701554 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.701559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.701563 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.701759 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.701928 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.702558 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.702672 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:22:35.703203 32719 sys_catalog.cc:455] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.703325 32719 sys_catalog.cc:458] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.703495 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.692218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.18.172.124:46719)
0924 20:22:35.692314 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.692318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.692387 (+ 69us) server_negotiation.cc:407] Connection header received
0924 20:22:35.692490 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.692494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.692531 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.693037 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.693128 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.693796 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.694784 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.695128 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.699244 (+ 4116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.699565 (+ 321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.699568 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.699571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.699597 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.701225 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.701688 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.701691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.701693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.701762 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.702022 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.702024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.702026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.702196 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.703158 (+ 962us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.703235 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.703357 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 20:22:35.706117 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:22:35.706467 32709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: Bootstrap starting.
I20260924 20:22:35.709173 32709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:35.709372 32633 raft_consensus.cc:348] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.709619 32633 raft_consensus.cc:374] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:35.709678 32633 raft_consensus.cc:729] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df46b1a419a04474be236274a188d879, State: Initialized, Role: FOLLOWER
I20260924 20:22:35.710070 32709 log.cc:824] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:35.710172 32633 consensus_queue.cc:260] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.711069 32737 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.711252 32737 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.711493 32633 sys_catalog.cc:564] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:35.712301 32709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: No bootstrap required, opened a new log
W20260924 20:22:35.716801 32748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:22:35.716920 32748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:22:35.717338 32709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.717600 32709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:35.717680 32709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44308ce7ada843eab93ecfd9be252f94, State: Initialized, Role: FOLLOWER
I20260924 20:22:35.718169 32709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.719983 32709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:35.723218 32753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.723349 32753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.719828 32565 master_runner.cc:186] Error getting master registration for 127.18.172.124:46719: OK, instance_id { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" instance_seqno: 1790281355651427 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:22:35.723528 32565 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:22:35.727501 32641 master_runner.cc:263] No leader master found from master 44308ce7ada843eab93ecfd9be252f94
I20260924 20:22:35.727574 32641 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 20:22:35.729099 32764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:22:35.729176 32764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:22:35.729739 32737 raft_consensus.cc:482] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:35.729889 32737 raft_consensus.cc:504] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.731189 32737 leader_election.cc:288] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af26a6c076374b38b2e4be807ff30e40 (127.18.172.126:41545), 44308ce7ada843eab93ecfd9be252f94 (127.18.172.124:46719)
I20260924 20:22:35.731622 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df46b1a419a04474be236274a188d879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44308ce7ada843eab93ecfd9be252f94" is_pre_election: true
I20260924 20:22:35.731688 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df46b1a419a04474be236274a188d879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af26a6c076374b38b2e4be807ff30e40" is_pre_election: true
I20260924 20:22:35.731932 32683 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df46b1a419a04474be236274a188d879 in term 0.
I20260924 20:22:35.731966 32532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df46b1a419a04474be236274a188d879 in term 0.
I20260924 20:22:35.732357 32581 leader_election.cc:302] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44308ce7ada843eab93ecfd9be252f94, df46b1a419a04474be236274a188d879; no voters:
I20260924 20:22:35.732807 32737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:22:35.732933 32737 raft_consensus.cc:482] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:22:35.732993 32737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:35.734361 32737 raft_consensus.cc:504] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.735059 32737 leader_election.cc:288] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [CANDIDATE]: Term 1 election: Requested vote from peers af26a6c076374b38b2e4be807ff30e40 (127.18.172.126:41545), 44308ce7ada843eab93ecfd9be252f94 (127.18.172.124:46719)
I20260924 20:22:35.735435 32683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df46b1a419a04474be236274a188d879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44308ce7ada843eab93ecfd9be252f94"
I20260924 20:22:35.735558 32683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:35.737108 32532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "df46b1a419a04474be236274a188d879" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af26a6c076374b38b2e4be807ff30e40"
I20260924 20:22:35.737241 32532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:35.737514 32683 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df46b1a419a04474be236274a188d879 in term 1.
I20260924 20:22:35.737807 32581 leader_election.cc:302] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44308ce7ada843eab93ecfd9be252f94, df46b1a419a04474be236274a188d879; no voters:
I20260924 20:22:35.737967 32737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:35.738155 32737 raft_consensus.cc:686] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [term 1 LEADER]: Becoming Leader. State: Replica: df46b1a419a04474be236274a188d879, State: Running, Role: LEADER
I20260924 20:22:35.738494 32737 consensus_queue.cc:237] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } }
I20260924 20:22:35.738756 32532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df46b1a419a04474be236274a188d879 in term 1.
I20260924 20:22:35.740043 32639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.718115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.18.172.125:39355)
0924 20:22:35.718218 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.718223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.730411 (+ 12188us) server_negotiation.cc:407] Connection header received
0924 20:22:35.730573 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.730576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.730621 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.730667 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.731578 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.732247 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.733720 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.734046 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.734114 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.736217 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.736221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.736224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.736249 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.737931 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.738571 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.738578 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.738583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.738681 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.739121 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.739124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.739126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.739340 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.739448 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.739782 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.739892 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:22:35.740273 32737 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df46b1a419a04474be236274a188d879. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.740381 32737 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:35.745890 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.717396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51896 (local address 127.18.172.126:41545)
0924 20:22:35.717528 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.717533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.722183 (+ 4650us) server_negotiation.cc:407] Connection header received
0924 20:22:35.722405 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.722409 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.722480 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.722536 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.724031 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.724826 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.726806 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.727164 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.727244 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.729098 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.729102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.729105 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.729136 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.730819 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.742464 (+ 11645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.742468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.742470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.742565 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.744858 (+ 2293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.744862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.744864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.745114 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.745254 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.745586 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.745717 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 20:22:35.748626 300 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:35.749636 300 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:35.752605 32715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:35.718475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.18.172.124:46719)
0924 20:22:35.718570 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:35.718576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:35.730917 (+ 12341us) server_negotiation.cc:407] Connection header received
0924 20:22:35.730958 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:35.730961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:35.731023 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:35.731096 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:35.742907 (+ 11811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.743731 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.746676 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:35.747115 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:35.747208 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:35.748995 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:35.748999 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:35.749001 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:35.749028 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:35.750720 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.751244 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.751248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.751250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.751331 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:35.751636 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:35.751639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:35.751641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:35.751848 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:35.751963 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:35.752312 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 20:22:35.752429 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 20:22:35.765666 32683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [term 1 FOLLOWER]: Refusing update from remote peer df46b1a419a04474be236274a188d879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:35.765691 32532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [term 1 FOLLOWER]: Refusing update from remote peer df46b1a419a04474be236274a188d879: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:35.766288 32766 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:35.766522 32737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:35.773910 32719 sys_catalog.cc:455] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df46b1a419a04474be236274a188d879. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.774034 32719 sys_catalog.cc:458] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.774906 32753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df46b1a419a04474be236274a188d879. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.775050 32753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.781404 32766 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.781530 32766 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:35.781642 301 mvcc.cc:205] Tried to move back new op lower bound from 7332992433202577408 to 7332992433108066304. Current Snapshot: MvccSnapshot[applied={T|T < 7332992433202577408}]
I20260924 20:22:35.782918 32719 sys_catalog.cc:455] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.783056 32719 sys_catalog.cc:458] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.787526 303 mvcc.cc:205] Tried to move back new op lower bound from 7332992433202577408 to 7332992433108066304. Current Snapshot: MvccSnapshot[applied={T|T < 7332992433202577408}]
I20260924 20:22:35.788830 32737 sys_catalog.cc:455] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.788947 32737 sys_catalog.cc:458] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:35.790578 32753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df46b1a419a04474be236274a188d879" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af26a6c076374b38b2e4be807ff30e40" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41545 } } peers { permanent_uuid: "df46b1a419a04474be236274a188d879" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 39355 } } peers { permanent_uuid: "44308ce7ada843eab93ecfd9be252f94" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 46719 } } }
I20260924 20:22:35.790684 32753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:22:35.791206 302 mvcc.cc:205] Tried to move back new op lower bound from 7332992433202577408 to 7332992433108066304. Current Snapshot: MvccSnapshot[applied={T|T < 7332992433202577408}]
I20260924 20:22:35.793203 300 catalog_manager.cc:1347] Generated new cluster ID: f88351003e5e446d921b0c613eff53d4
I20260924 20:22:35.793274 300 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:35.816597 300 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:22:35.817988 300 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:35.834045 300 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df46b1a419a04474be236274a188d879: Generated new TSK 0
I20260924 20:22:35.835171 300 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:22:36.700455 32730 catalog_manager.cc:1259] Loaded cluster ID: f88351003e5e446d921b0c613eff53d4
I20260924 20:22:36.700585 32730 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: loading cluster ID for follower catalog manager: success
I20260924 20:22:36.702447 32730 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: acquiring CA information for follower catalog manager: success
I20260924 20:22:36.703768 32730 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af26a6c076374b38b2e4be807ff30e40: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:22:36.730545 32764 catalog_manager.cc:1259] Loaded cluster ID: f88351003e5e446d921b0c613eff53d4
I20260924 20:22:36.730639 32764 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: loading cluster ID for follower catalog manager: success
I20260924 20:22:36.732384 32764 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: acquiring CA information for follower catalog manager: success
I20260924 20:22:36.733548 32764 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44308ce7ada843eab93ecfd9be252f94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:22:36.896113 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 32490
I20260924 20:22:36.902882 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 32565
I20260924 20:22:36.908766 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 32641
2026-09-24T20:22:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3112 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:22:36 dist-test-slave-m1qq krb5kdc[315](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:22:36 dist-test-slave-m1qq krb5kdc[315](info): set up 2 sockets
Sep 24 20:22:36 dist-test-slave-m1qq krb5kdc[315](info): commencing operation
krb5kdc: starting...
W20260924 20:22:38.004923 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:22:38.130439 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39121
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:38.253242 328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:38.253554 328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:38.253597 328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:38.257932 328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:38.257997 328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:38.258016 328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:38.258034 328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:38.258050 328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:38.263373 328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:38.264402 328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:38.265367 328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:38.274055 334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:38.274084 333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:38.274097 328 server_base.cc:1034] running on GCE node
W20260924 20:22:38.274060 336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:38.275081 328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:38.276387 328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:38.277603 328 hybrid_clock.cc:648] HybridClock initialized: now 1790281358277583 us; error 32 us; skew 500 ppm
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[315](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281358, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:38.280794 328 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:38.281803 328 webserver.cc:460] Webserver started at http://127.18.172.126:44517/ using document root <none> and password file <none>
I20260924 20:22:38.282433 328 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:38.282536 328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:38.282800 328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:38.285090 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "0045878900c64c5db6008cb8efec87a4"
format_stamp: "Formatted at 2026-09-24 20:22:38 on dist-test-slave-m1qq"
server_key: "e9400f06b23cf031f4d0bbc2f3a8e833"
server_key_iv: "55516a85bca571ef6499f1c0753d649f"
server_key_version: "encryptionkey@0"
I20260924 20:22:38.285710 328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "0045878900c64c5db6008cb8efec87a4"
format_stamp: "Formatted at 2026-09-24 20:22:38 on dist-test-slave-m1qq"
server_key: "e9400f06b23cf031f4d0bbc2f3a8e833"
server_key_iv: "55516a85bca571ef6499f1c0753d649f"
server_key_version: "encryptionkey@0"
I20260924 20:22:38.289889 328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260924 20:22:38.292686 343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:38.294020 328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:22:38.294180 328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "0045878900c64c5db6008cb8efec87a4"
format_stamp: "Formatted at 2026-09-24 20:22:38 on dist-test-slave-m1qq"
server_key: "e9400f06b23cf031f4d0bbc2f3a8e833"
server_key_iv: "55516a85bca571ef6499f1c0753d649f"
server_key_version: "encryptionkey@0"
I20260924 20:22:38.294307 328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:38.309859 328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:38.313084 328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:38.313283 328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:38.320734 328 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:39121
I20260924 20:22:38.320770 395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:39121 every 8 connection(s)
I20260924 20:22:38.321678 328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:38.324416 396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:38.326993 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 328
I20260924 20:22:38.327183 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:22:38.327478 19121 external_mini_cluster.cc:1428] Setting key c36a252c9816da1bdefa91e8d982c219
I20260924 20:22:38.330533 396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4: Bootstrap starting.
I20260924 20:22:38.333170 396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:38.334031 396 log.cc:824] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:38.336200 396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4: No bootstrap required, opened a new log
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[315](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:38.340323 396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } }
I20260924 20:22:38.340606 396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:38.340660 396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0045878900c64c5db6008cb8efec87a4, State: Initialized, Role: FOLLOWER
I20260924 20:22:38.341202 396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } }
I20260924 20:22:38.341382 396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:38.341470 396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:38.341574 396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:38.342307 399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:38.328800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.18.172.126:39121)
0924 20:22:38.329201 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:38.329221 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:38.329257 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:22:38.329926 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:38.329943 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:38.330281 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:38.330562 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:38.331541 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:38.332562 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:38.333735 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:38.334258 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:38.334375 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:38.337207 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:38.337212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:38.337225 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:38.337251 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:38.339125 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:38.339607 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:38.339612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:38.339617 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:38.339722 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:38.340010 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:38.340012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:38.340013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:38.340393 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:38.340552 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:38.341027 (+ 475us) server_negotiation.cc:299] Negotiation successful
0924 20:22:38.341306 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":118,"threads_started":1}
I20260924 20:22:38.342567 396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } }
I20260924 20:22:38.342926 396 leader_election.cc:302] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0045878900c64c5db6008cb8efec87a4; no voters:
I20260924 20:22:38.343302 396 leader_election.cc:288] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:22:38.343544 401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:38.343927 401 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [term 1 LEADER]: Becoming Leader. State: Replica: 0045878900c64c5db6008cb8efec87a4, State: Running, Role: LEADER
I20260924 20:22:38.344329 401 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } }
I20260924 20:22:38.344611 396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:38.345609 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:22:38.346145 402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0045878900c64c5db6008cb8efec87a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } } }
I20260924 20:22:38.346292 402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:38.346722 403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0045878900c64c5db6008cb8efec87a4. Latest consensus state: current_term: 1 leader_uuid: "0045878900c64c5db6008cb8efec87a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0045878900c64c5db6008cb8efec87a4" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39121 } } }
I20260924 20:22:38.346877 403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:38.347513 413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:38.350296 413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:38.355540 413 catalog_manager.cc:1347] Generated new cluster ID: b8177f9afe8a40e78fd96d6d990ea554
I20260924 20:22:38.355619 413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:38.380436 413 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:22:38.381443 413 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:38.399717 413 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0045878900c64c5db6008cb8efec87a4: Generated new TSK 0
I20260924 20:22:38.400444 413 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:22:38.453231 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 328
2026-09-24T20:22:38Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1542 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[423](info): set up 2 sockets
Sep 24 20:22:38 dist-test-slave-m1qq krb5kdc[423](info): commencing operation
krb5kdc: starting...
W20260924 20:22:39.604363 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:22:39 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:22:39.732534 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:39.845228 436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:39.845518 436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:39.845587 436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:39.849558 436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:39.849644 436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:39.849682 436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:39.849733 436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:39.849766 436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:39.855054 436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:45629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:39.856081 436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:39.857118 436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:39.865700 441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:39.865948 436 server_base.cc:1034] running on GCE node
W20260924 20:22:39.865717 444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:39.865700 442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:39.866590 436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:39.867763 436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:39.868920 436 hybrid_clock.cc:648] HybridClock initialized: now 1790281359868897 us; error 42 us; skew 500 ppm
Sep 24 20:22:39 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281359, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:39.871812 436 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:39.872658 436 webserver.cc:460] Webserver started at http://127.18.172.126:46289/ using document root <none> and password file <none>
I20260924 20:22:39.873205 436 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:39.873265 436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:39.873556 436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:39.875653 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "37f81d4e27c74b5f98d4df2f9c175d52"
format_stamp: "Formatted at 2026-09-24 20:22:39 on dist-test-slave-m1qq"
server_key: "c2214fde8f3e6c7ecfa777d3d4584de3"
server_key_iv: "b2c2e3b9772b650022136486001e5da9"
server_key_version: "encryptionkey@0"
I20260924 20:22:39.876219 436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "37f81d4e27c74b5f98d4df2f9c175d52"
format_stamp: "Formatted at 2026-09-24 20:22:39 on dist-test-slave-m1qq"
server_key: "c2214fde8f3e6c7ecfa777d3d4584de3"
server_key_iv: "b2c2e3b9772b650022136486001e5da9"
server_key_version: "encryptionkey@0"
I20260924 20:22:39.880085 436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:22:39.882370 451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:39.883708 436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:22:39.883850 436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "37f81d4e27c74b5f98d4df2f9c175d52"
format_stamp: "Formatted at 2026-09-24 20:22:39 on dist-test-slave-m1qq"
server_key: "c2214fde8f3e6c7ecfa777d3d4584de3"
server_key_iv: "b2c2e3b9772b650022136486001e5da9"
server_key_version: "encryptionkey@0"
I20260924 20:22:39.883966 436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:39.898361 436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:39.901662 436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:39.901880 436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:39.909165 436 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:45629
I20260924 20:22:39.909492 503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:45629 every 8 connection(s)
I20260924 20:22:39.910147 436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:39.912976 504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:39.918382 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap starting.
I20260924 20:22:39.919275 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 436
I20260924 20:22:39.919427 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:22:39.919772 19121 external_mini_cluster.cc:1428] Setting key e80b65f4a5144654e58d5df9fe7267c9
I20260924 20:22:39.921315 504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:39.922008 504 log.cc:824] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:39.924206 504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: No bootstrap required, opened a new log
I20260924 20:22:39.928598 504 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:39.928821 504 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:39.928876 504 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Initialized, Role: FOLLOWER
Sep 24 20:22:39 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:39.929378 504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:39.929546 504 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:39.929620 504 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:39.929734 504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:39.930620 504 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:39.930958 504 leader_election.cc:302] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f81d4e27c74b5f98d4df2f9c175d52; no voters:
I20260924 20:22:39.931322 504 leader_election.cc:288] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:22:39.931453 509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:39.931682 509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 LEADER]: Becoming Leader. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Running, Role: LEADER
I20260924 20:22:39.932078 509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:39.932600 504 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:39.935071 510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:39.935189 510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:39.935640 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:39.920950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54744 (local address 127.18.172.126:45629)
0924 20:22:39.921369 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:39.921400 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:39.921452 (+ 52us) server_negotiation.cc:407] Connection header received
0924 20:22:39.922089 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:39.922114 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:39.922634 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:39.922950 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:39.923878 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:39.924941 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:39.926168 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:39.926636 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:39.926752 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:39.929765 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:39.929770 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:39.929782 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:39.929833 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:39.932104 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:39.932616 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:39.932621 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:39.932628 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:39.932766 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:39.933087 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:39.933090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:39.933092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:39.933649 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:39.933835 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:39.934228 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 20:22:39.934538 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":150,"threads_started":1}
I20260924 20:22:39.935616 511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f81d4e27c74b5f98d4df2f9c175d52. Latest consensus state: current_term: 1 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:39.935949 511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
W20260924 20:22:39.937896 523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:22:39.937971 523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:22:39.938025 525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:39.938871 525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:39.944134 525 catalog_manager.cc:1347] Generated new cluster ID: 6751290c378c4d57a639a5008cccb1ff
I20260924 20:22:39.944195 525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:39.956686 525 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:22:39.957680 525 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:39.962553 525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Generated new TSK 0
I20260924 20:22:39.963279 525 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:22:40.000247 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:40.116670 530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:40.116912 530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:40.116961 530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:40.120875 530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:40.120934 530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:40.120994 530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:40.126092 530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:40.127187 530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:40.128242 530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:40.138935 535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:40.138931 538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:40.139149 536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:40.139307 530 server_base.cc:1034] running on GCE node
I20260924 20:22:40.139750 530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:40.140347 530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:40.141621 530 hybrid_clock.cc:648] HybridClock initialized: now 1790281360141588 us; error 41 us; skew 500 ppm
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:40.144788 530 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:40.145679 530 webserver.cc:460] Webserver started at http://127.18.172.65:46839/ using document root <none> and password file <none>
I20260924 20:22:40.146224 530 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:40.146284 530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:40.146457 530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:40.148631 530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "1b2912cc83d344659f219f20e30df1f8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "377609370b464631d7fde29d514d0f61"
server_key_iv: "ba785115e3b5041b1708299f099eb2b2"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.149222 530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "1b2912cc83d344659f219f20e30df1f8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "377609370b464631d7fde29d514d0f61"
server_key_iv: "ba785115e3b5041b1708299f099eb2b2"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.153200 530 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260924 20:22:40.155632 545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.157099 530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:22:40.157235 530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "1b2912cc83d344659f219f20e30df1f8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "377609370b464631d7fde29d514d0f61"
server_key_iv: "ba785115e3b5041b1708299f099eb2b2"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.157343 530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:40.182682 530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:40.186144 530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:40.186350 530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:40.187069 530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:40.187896 530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:40.187973 530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.188026 530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:40.188062 530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.197362 530 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:32847
I20260924 20:22:40.197453 658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:32847 every 8 connection(s)
I20260924 20:22:40.198290 530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:40.207538 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 530
I20260924 20:22:40.207675 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:40.208011 19121 external_mini_cluster.cc:1428] Setting key 1d5c231d216c6c1bfdd7c8b77b67254b
I20260924 20:22:40.212437 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.200203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:52539 (local address 127.18.172.126:45629)
0924 20:22:40.200374 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.200379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.201228 (+ 849us) server_negotiation.cc:407] Connection header received
0924 20:22:40.202566 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.202568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.202615 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.202693 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:40.203911 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.204481 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.205582 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.205804 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.205880 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.208463 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:40.208466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:40.208468 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:40.208492 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:40.210229 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.210768 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.210775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.210780 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.210875 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.211335 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.211340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.211342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.211527 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:40.211637 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.212137 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.212268 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:22:40.213258 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.200591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:52539)
0924 20:22:40.201072 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.201116 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.202364 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.202852 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.202858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.203298 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:40.203780 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.203792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.204609 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.204612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.205475 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.205484 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.205931 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.205933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.206095 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.206633 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:40.206649 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:40.208279 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:40.210352 (+ 2073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.210356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.210367 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.210665 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.211052 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.211059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.211061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.211234 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.211634 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:40.211640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:40.211872 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.212082 (+ 210us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.212402 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":107,"threads_started":1}
I20260924 20:22:40.214608 659 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:40.214845 659 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:40.215366 659 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:40.216828 468 ts_manager.cc:194] Registered new tserver with Master: 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847)
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:22:40.217970 468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:52539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:22:40.238672 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:40.352831 664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:40.353160 664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:40.353240 664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:40.357311 664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:40.357378 664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:40.357475 664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:22:40.363211 664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:40.364308 664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:40.365325 664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:40.375314 672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:40.375355 664 server_base.cc:1034] running on GCE node
W20260924 20:22:40.375492 670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:40.375363 669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:40.376113 664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:40.376806 664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:40.378046 664 hybrid_clock.cc:648] HybridClock initialized: now 1790281360378023 us; error 43 us; skew 500 ppm
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:40.381026 664 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:22:40.382012 664 webserver.cc:460] Webserver started at http://127.18.172.66:45865/ using document root <none> and password file <none>
I20260924 20:22:40.382618 664 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:40.382717 664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:40.382964 664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:40.385147 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "a3a8a78c658a493ebc3d524249cfc30d"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "f0614a34bf29ab3c8fb8e42bf7b1ea1f"
server_key_iv: "6356e003cf6dd0a77520b1bbf5958785"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.385725 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "a3a8a78c658a493ebc3d524249cfc30d"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "f0614a34bf29ab3c8fb8e42bf7b1ea1f"
server_key_iv: "6356e003cf6dd0a77520b1bbf5958785"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.389783 664 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260924 20:22:40.392547 679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.393946 664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:22:40.394100 664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a3a8a78c658a493ebc3d524249cfc30d"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "f0614a34bf29ab3c8fb8e42bf7b1ea1f"
server_key_iv: "6356e003cf6dd0a77520b1bbf5958785"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.394223 664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:40.416903 664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:40.420091 664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:40.420313 664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:40.420924 664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:40.421780 664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:40.421854 664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.421928 664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:40.421972 664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.431198 664 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:34601
I20260924 20:22:40.431288 792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:34601 every 8 connection(s)
I20260924 20:22:40.432371 664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:22:40.435694 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 664
I20260924 20:22:40.435818 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:22:40.436089 19121 external_mini_cluster.cc:1428] Setting key da4b601e95038116a592ce01dd9bc035
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:22:40.448067 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.434456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48653 (local address 127.18.172.126:45629)
0924 20:22:40.434618 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.434622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.435722 (+ 1100us) server_negotiation.cc:407] Connection header received
0924 20:22:40.437254 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.437258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.437312 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.437386 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:40.438544 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.439382 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.440539 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.440953 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.441095 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.443753 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:40.443756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:40.443759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:40.443782 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:40.445576 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.446032 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.446035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.446036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.446108 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.446510 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.446514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.446516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.446766 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:40.447041 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.447716 (+ 675us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.447838 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:22:40.448935 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.434960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:48653)
0924 20:22:40.435515 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.435621 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.437034 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.437553 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.437561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.437915 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:40.438383 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.438410 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.439523 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.439526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.440405 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.440415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.441110 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.441135 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.441333 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.441959 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:40.441974 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:40.443581 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:40.445687 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.445692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.445704 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.445948 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.446188 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.446190 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.446192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.446377 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.447042 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:40.447049 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:40.447314 (+ 265us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.447660 (+ 346us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.447989 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260924 20:22:40.450215 793 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:40.450466 793 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:40.450955 793 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:40.452091 468 ts_manager.cc:194] Registered new tserver with Master: a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601)
I20260924 20:22:40.453011 468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:48653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:22:40.466246 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:40.576622 798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:40.576901 798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:40.576948 798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:40.581012 798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:40.581072 798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:40.581132 798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:22:40.586362 798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:40.587576 798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:40.588601 798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:40.598212 804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:40.598212 803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:40.598260 806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:40.598562 798 server_base.cc:1034] running on GCE node
I20260924 20:22:40.598953 798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:40.599604 798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:40.600780 798 hybrid_clock.cc:648] HybridClock initialized: now 1790281360600741 us; error 47 us; skew 500 ppm
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:40.604017 798 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:22:40.604967 798 webserver.cc:460] Webserver started at http://127.18.172.67:33949/ using document root <none> and password file <none>
I20260924 20:22:40.605537 798 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:40.605597 798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:40.605808 798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:40.607900 798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "21f71bb592894c649988350dd5af12a8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "3d9e0936fc6912cb8b24a3a525c0c4a3"
server_key_iv: "e6d27d21f8fe3c45fcd0cda10a3f98f6"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.608470 798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "21f71bb592894c649988350dd5af12a8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "3d9e0936fc6912cb8b24a3a525c0c4a3"
server_key_iv: "e6d27d21f8fe3c45fcd0cda10a3f98f6"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.612411 798 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:22:40.614941 813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.616362 798 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:22:40.616585 798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "21f71bb592894c649988350dd5af12a8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "3d9e0936fc6912cb8b24a3a525c0c4a3"
server_key_iv: "e6d27d21f8fe3c45fcd0cda10a3f98f6"
server_key_version: "encryptionkey@0"
I20260924 20:22:40.616709 798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:40.629837 798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:40.633417 798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:40.633649 798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:40.634253 798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:40.635176 798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:40.635248 798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.635320 798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:40.635362 798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:40.644050 798 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42773
I20260924 20:22:40.644069 926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42773 every 8 connection(s)
I20260924 20:22:40.644941 798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:22:40.652614 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 798
I20260924 20:22:40.652760 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:22:40.653103 19121 external_mini_cluster.cc:1428] Setting key 17b4231cd64338e1a10e898f0feaee89
Sep 24 20:22:40 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281360, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:40.658881 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.646700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:44135 (local address 127.18.172.126:45629)
0924 20:22:40.646863 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.646868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.647747 (+ 879us) server_negotiation.cc:407] Connection header received
0924 20:22:40.649087 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.649089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.649142 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.649218 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:40.650369 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.650893 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.652066 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.652285 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.652380 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.655645 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:40.655649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:40.655652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:40.655679 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:40.656899 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.657385 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.657387 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.657389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.657461 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.657831 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.657833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.657834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.658000 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:40.658074 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.658670 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.658765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:22:40.659845 929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.647160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:44135)
0924 20:22:40.647573 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.647636 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.648878 (+ 1242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.649380 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.649386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.649769 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:40.650233 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.650245 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.651018 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.651020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.651948 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.651957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.652455 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.652459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.652703 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.653397 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:40.653420 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:40.655344 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:40.657016 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.657021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.657032 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.657287 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.657571 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:40.657574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:40.657576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:40.657733 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:40.658093 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:40.658098 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:40.658336 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.658592 (+ 256us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.658868 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":104,"threads_started":1}
I20260924 20:22:40.661160 927 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:40.661424 927 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:40.661950 927 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:40.663055 468 ts_manager.cc:194] Registered new tserver with Master: 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
I20260924 20:22:40.663712 468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:44135
I20260924 20:22:40.667573 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:22:40.676898 507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.669584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54754 (local address 127.18.172.126:45629)
0924 20:22:40.669700 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.669704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.669928 (+ 224us) server_negotiation.cc:407] Connection header received
0924 20:22:40.670031 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.670033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.670071 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.670140 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:40.670836 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.671402 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.672268 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.672549 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.672665 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.673898 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:40.673900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:40.673902 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:40.673922 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:40.675422 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.675792 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.675795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.675796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.675866 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:40.676112 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:40.676114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:40.676115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:40.676307 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:40.676427 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.676702 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.676787 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:22:40.679738 468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:22:40.682677 468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:22:40.697196 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.691592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51126 (local address 127.18.172.65:32847)
0924 20:22:40.692099 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.692114 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.692249 (+ 135us) server_negotiation.cc:407] Connection header received
0924 20:22:40.692914 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.692920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.693096 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.693285 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:40.693700 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694787 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.695908 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.696598 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.696706 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.696848 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.696940 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.697027 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":95,"threads_started":1}
I20260924 20:22:40.697232 938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.690815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:32847 (local address 127.0.0.1:51126)
0924 20:22:40.692052 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.692107 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.692252 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.693460 (+ 1208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.693464 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.693485 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:40.693566 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.693573 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694939 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.694954 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.695753 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.695766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.696726 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.696730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.696822 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.696845 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.696920 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.697081 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1105,"spinlock_wait_cycles":5760,"thread_start_us":84,"threads_started":1}
I20260924 20:22:40.697198 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.690912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.18.172.67:42773)
0924 20:22:40.691600 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.691613 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.692191 (+ 578us) server_negotiation.cc:407] Connection header received
0924 20:22:40.692382 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.692386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.692555 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.692706 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:40.692944 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694000 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.695800 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.696394 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.696508 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.696621 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.696899 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.697031 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":140,"threads_started":1}
I20260924 20:22:40.697706 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.690791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42773 (local address 127.0.0.1:60422)
0924 20:22:40.692052 (+ 1261us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.692107 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.692252 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.692680 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.692688 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.692715 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:40.692821 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.692826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694150 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.694153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.695666 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.695677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.696542 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.696546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.696668 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.696720 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.697490 (+ 770us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.697569 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1090,"mutex_wait_us":25,"spinlock_wait_cycles":6656,"thread_start_us":111,"threads_started":1}
I20260924 20:22:40.698593 939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.691084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34601 (local address 127.0.0.1:36598)
0924 20:22:40.692481 (+ 1397us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.692491 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.692597 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.693089 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.693092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.693110 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:40.693203 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.693209 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694420 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.694422 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.695668 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.695677 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.698180 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.698183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.698267 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.698290 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.698371 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.698438 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":41,"thread_start_us":84,"threads_started":1}
I20260924 20:22:40.699198 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.691592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36598 (local address 127.18.172.66:34601)
0924 20:22:40.691901 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.691913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.692586 (+ 673us) server_negotiation.cc:407] Connection header received
0924 20:22:40.692745 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.692760 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.692938 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.693094 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:40.693299 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.694260 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.696701 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.698044 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.698163 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.698293 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.698867 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.698966 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":99,"threads_started":1}
I20260924 20:22:40.700677 861 tablet_service.cc:1467] Processing CreateTablet for tablet d34df94451464229a286414fa4f2e1bd (DEFAULT_TABLE table=test-table [id=1a678cd6cd4b4d6a8ac23803da63f204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:40.700701 593 tablet_service.cc:1467] Processing CreateTablet for tablet d34df94451464229a286414fa4f2e1bd (DEFAULT_TABLE table=test-table [id=1a678cd6cd4b4d6a8ac23803da63f204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:40.701684 861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34df94451464229a286414fa4f2e1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:40.701696 593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34df94451464229a286414fa4f2e1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:40.701686 727 tablet_service.cc:1467] Processing CreateTablet for tablet d34df94451464229a286414fa4f2e1bd (DEFAULT_TABLE table=test-table [id=1a678cd6cd4b4d6a8ac23803da63f204]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:40.702659 727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34df94451464229a286414fa4f2e1bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:40.707963 944 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Bootstrap starting.
I20260924 20:22:40.709265 945 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Bootstrap starting.
I20260924 20:22:40.710110 946 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Bootstrap starting.
I20260924 20:22:40.710322 944 tablet_bootstrap.cc:654] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:40.711400 945 tablet_bootstrap.cc:654] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:40.711501 944 log.cc:824] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:40.712635 946 tablet_bootstrap.cc:654] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:40.712695 945 log.cc:824] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:40.713376 946 log.cc:824] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:40.713925 944 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: No bootstrap required, opened a new log
I20260924 20:22:40.714176 944 ts_tablet_manager.cc:1397] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:22:40.714643 945 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: No bootstrap required, opened a new log
I20260924 20:22:40.714768 945 ts_tablet_manager.cc:1397] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:22:40.715334 946 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: No bootstrap required, opened a new log
I20260924 20:22:40.715521 946 ts_tablet_manager.cc:1397] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:22:40.717729 945 raft_consensus.cc:348] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.717986 945 raft_consensus.cc:374] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:40.718060 945 raft_consensus.cc:729] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2912cc83d344659f219f20e30df1f8, State: Initialized, Role: FOLLOWER
I20260924 20:22:40.718585 945 consensus_queue.cc:260] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.719051 944 raft_consensus.cc:348] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.719249 659 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
I20260924 20:22:40.719298 944 raft_consensus.cc:374] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:40.719195 946 raft_consensus.cc:348] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.719398 944 raft_consensus.cc:729] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f71bb592894c649988350dd5af12a8, State: Initialized, Role: FOLLOWER
I20260924 20:22:40.719468 946 raft_consensus.cc:374] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:40.719530 946 raft_consensus.cc:729] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a8a78c658a493ebc3d524249cfc30d, State: Initialized, Role: FOLLOWER
I20260924 20:22:40.719866 944 consensus_queue.cc:260] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.719937 946 consensus_queue.cc:260] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.720836 945 ts_tablet_manager.cc:1428] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:22:40.720836 944 ts_tablet_manager.cc:1428] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:22:40.721086 946 ts_tablet_manager.cc:1428] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:22:40.721170 927 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
I20260924 20:22:40.721398 793 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
I20260924 20:22:40.869525 952 raft_consensus.cc:482] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:40.869779 952 raft_consensus.cc:504] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.871060 952 leader_election.cc:288] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773), 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847)
I20260924 20:22:40.875406 795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.871279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42773 (local address 127.18.172.66:33731)
0924 20:22:40.871489 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.871509 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.871650 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.871945 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.871948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.871966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:40.872079 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.872083 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.872828 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.872832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.873876 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.873891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.874902 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.874906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.875072 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.875094 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.875176 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.875253 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":47}
I20260924 20:22:40.875639 953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.871362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:32847 (local address 127.18.172.66:49529)
0924 20:22:40.871750 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.871767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:40.871907 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:40.872200 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:40.872203 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:40.872218 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:40.872337 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.872342 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.873049 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.873053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.874114 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:40.874125 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.875193 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.875196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:40.875327 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.875348 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:22:40.875422 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:22:40.875492 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":45,"thread_start_us":165,"threads_started":1}
I20260924 20:22:40.875890 940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.871390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:33731 (local address 127.18.172.67:42773)
0924 20:22:40.871537 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.871543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.871688 (+ 145us) server_negotiation.cc:407] Connection header received
0924 20:22:40.871756 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.871759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.871813 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.871900 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:40.872195 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.872711 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.874035 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.874759 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.874901 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.874959 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.875685 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.875772 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:22:40.876192 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.871585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:49529 (local address 127.18.172.65:32847)
0924 20:22:40.871727 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.871732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.871855 (+ 123us) server_negotiation.cc:407] Connection header received
0924 20:22:40.872049 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.872051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.872099 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.872180 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:40.872444 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.872941 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.874280 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.875065 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.875200 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.875262 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.875485 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.876027 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:22:40.876565 881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34df94451464229a286414fa4f2e1bd" candidate_uuid: "a3a8a78c658a493ebc3d524249cfc30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f71bb592894c649988350dd5af12a8" is_pre_election: true
I20260924 20:22:40.876973 881 raft_consensus.cc:2454] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a8a78c658a493ebc3d524249cfc30d in term 0.
I20260924 20:22:40.877571 682 leader_election.cc:302] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d; no voters:
I20260924 20:22:40.877926 952 raft_consensus.cc:2781] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:22:40.878055 952 raft_consensus.cc:482] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:22:40.878119 952 raft_consensus.cc:3037] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:40.879503 613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34df94451464229a286414fa4f2e1bd" candidate_uuid: "a3a8a78c658a493ebc3d524249cfc30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2912cc83d344659f219f20e30df1f8" is_pre_election: true
I20260924 20:22:40.879644 952 raft_consensus.cc:504] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.879814 613 raft_consensus.cc:2454] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3a8a78c658a493ebc3d524249cfc30d in term 0.
I20260924 20:22:40.880053 952 leader_election.cc:288] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [CANDIDATE]: Term 1 election: Requested vote from peers 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773), 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847)
I20260924 20:22:40.880401 881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34df94451464229a286414fa4f2e1bd" candidate_uuid: "a3a8a78c658a493ebc3d524249cfc30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f71bb592894c649988350dd5af12a8"
I20260924 20:22:40.880502 613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34df94451464229a286414fa4f2e1bd" candidate_uuid: "a3a8a78c658a493ebc3d524249cfc30d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b2912cc83d344659f219f20e30df1f8"
I20260924 20:22:40.880520 881 raft_consensus.cc:3037] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:40.880609 613 raft_consensus.cc:3037] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:40.881523 881 raft_consensus.cc:2454] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a8a78c658a493ebc3d524249cfc30d in term 1.
I20260924 20:22:40.881750 613 raft_consensus.cc:2454] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3a8a78c658a493ebc3d524249cfc30d in term 1.
I20260924 20:22:40.881829 682 leader_election.cc:302] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d; no voters:
I20260924 20:22:40.882007 952 raft_consensus.cc:2781] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:40.882236 952 raft_consensus.cc:686] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 1 LEADER]: Becoming Leader. State: Replica: a3a8a78c658a493ebc3d524249cfc30d, State: Running, Role: LEADER
I20260924 20:22:40.882573 952 consensus_queue.cc:237] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:40.885933 468 catalog_manager.cc:5495] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d reported cstate change: term changed from 0 to 1, leader changed from <none> to a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66). New cstate: current_term: 1 leader_uuid: "a3a8a78c658a493ebc3d524249cfc30d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } health_report { overall_health: HEALTHY } } }
W20260924 20:22:40.895874 928 tablet.cc:2369] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:22:40.933526 794 tablet.cc:2369] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:22:40.938375 943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.934580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.18.172.66:34601)
0924 20:22:40.934708 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:40.934715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:40.934732 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:22:40.934799 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:40.934802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:40.934854 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:40.934949 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:22:40.935292 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.935896 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.936821 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:40.937041 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:40.937125 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:40.937527 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:22:40.937944 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:22:40.938035 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:40.938128 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 20:22:40.938181 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:22:40.944187 613 raft_consensus.cc:1261] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Refusing update from remote peer a3a8a78c658a493ebc3d524249cfc30d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:40.944186 881 raft_consensus.cc:1261] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Refusing update from remote peer a3a8a78c658a493ebc3d524249cfc30d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:40.944888 954 consensus_queue.cc:1035] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:40.945134 952 consensus_queue.cc:1035] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:22:40.949520 660 tablet.cc:2369] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:22:40.961953 467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:40.962118 467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:40.964519 467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Sending DeleteTablet for 3 replicas of tablet d34df94451464229a286414fa4f2e1bd
I20260924 20:22:40.965310 861 tablet_service.cc:1514] Processing DeleteTablet for tablet d34df94451464229a286414fa4f2e1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:40 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:60422
I20260924 20:22:40.965303 727 tablet_service.cc:1514] Processing DeleteTablet for tablet d34df94451464229a286414fa4f2e1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:40 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:36598
I20260924 20:22:40.965365 593 tablet_service.cc:1514] Processing DeleteTablet for tablet d34df94451464229a286414fa4f2e1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:40 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51126
I20260924 20:22:40.965838 967 tablet_replica.cc:330] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: stopping tablet replica
I20260924 20:22:40.965888 966 tablet_replica.cc:330] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: stopping tablet replica
I20260924 20:22:40.966213 966 raft_consensus.cc:2229] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:40.966248 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 530
I20260924 20:22:40.966238 967 raft_consensus.cc:2229] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:40.967926 967 raft_consensus.cc:2258] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:40.968237 968 tablet_replica.cc:330] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: stopping tablet replica
I20260924 20:22:40.968962 968 raft_consensus.cc:2229] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:22:40.969432 968 raft_consensus.cc:2258] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:40.969722 967 ts_tablet_manager.cc:1905] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:40.970829 968 ts_tablet_manager.cc:1905] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:40.972644 967 ts_tablet_manager.cc:1918] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:40.972805 967 log.cc:1196] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/d34df94451464229a286414fa4f2e1bd
I20260924 20:22:40.973220 967 ts_tablet_manager.cc:1939] T d34df94451464229a286414fa4f2e1bd P 21f71bb592894c649988350dd5af12a8: Deleting consensus metadata
W20260924 20:22:40.973730 452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:32847 (error 108)
I20260924 20:22:40.973937 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 664
I20260924 20:22:40.975561 454 catalog_manager.cc:4841] TS 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): tablet d34df94451464229a286414fa4f2e1bd (table test-table [id=1a678cd6cd4b4d6a8ac23803da63f204]) successfully deleted
I20260924 20:22:40.976182 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.975919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:32847 (local address 127.0.0.1:51140)
0924 20:22:40.976052 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.976112 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 20:22:40.976372 452 catalog_manager.cc:4584] TS 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d34df94451464229a286414fa4f2e1bd: Network error: Client connection negotiation failed: client connection to 127.18.172.65:32847: connect: Connection refused (error 111)
I20260924 20:22:40.979473 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 798
I20260924 20:22:40.980317 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:40.979517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34601 (local address 127.0.0.1:36602)
0924 20:22:40.980154 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:40.980220 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:34601: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":532}
I20260924 20:22:40.985150 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 436
I20260924 20:22:40.991429 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--webserver_interface=127.18.172.126
--webserver_port=46289
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:41.102276 970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:41.102555 970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:41.102595 970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:41.107024 970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:41.107080 970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:41.107106 970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:41.107123 970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:41.107138 970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:41.112290 970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:45629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=46289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:41.113241 970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:41.114184 970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:41.127434 976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:41.127434 977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.127653 970 server_base.cc:1034] running on GCE node
W20260924 20:22:41.127708 979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.128403 970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:41.129635 970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:41.130842 970 hybrid_clock.cc:648] HybridClock initialized: now 1790281361130809 us; error 52 us; skew 500 ppm
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:41.133756 970 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:41.134688 970 webserver.cc:460] Webserver started at http://127.18.172.126:46289/ using document root <none> and password file <none>
I20260924 20:22:41.135304 970 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:41.135380 970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:41.139386 970 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:22:41.141433 986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.142840 970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:22:41.143029 970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "37f81d4e27c74b5f98d4df2f9c175d52"
format_stamp: "Formatted at 2026-09-24 20:22:39 on dist-test-slave-m1qq"
server_key: "c2214fde8f3e6c7ecfa777d3d4584de3"
server_key_iv: "b2c2e3b9772b650022136486001e5da9"
server_key_version: "encryptionkey@0"
I20260924 20:22:41.143698 970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:41.163058 970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:41.166107 970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:41.166316 970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:41.174206 970 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:45629
I20260924 20:22:41.174504 1038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:45629 every 8 connection(s)
I20260924 20:22:41.175377 970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:41.178020 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 970
I20260924 20:22:41.178562 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:32847
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=46839
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:41.179986 1039 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:22:41.181638 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap starting.
I20260924 20:22:41.199791 1039 log.cc:824] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:41.208312 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:22:41.208688 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap complete.
I20260924 20:22:41.212821 1039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:41.213222 1039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Initialized, Role: FOLLOWER
I20260924 20:22:41.213697 1039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:41.213846 1039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:41.213927 1039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:41.214047 1039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:22:41.216053 1039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:41.216379 1039 leader_election.cc:302] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f81d4e27c74b5f98d4df2f9c175d52; no voters:
I20260924 20:22:41.216719 1039 leader_election.cc:288] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:22:41.216801 1043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:22:41.216992 1043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 LEADER]: Becoming Leader. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Running, Role: LEADER
I20260924 20:22:41.217281 1043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:41.217963 1039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:41.218374 1043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f81d4e27c74b5f98d4df2f9c175d52. Latest consensus state: current_term: 2 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:41.218504 1043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:41.218863 1047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:41.218876 1044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:41.219130 1044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:41.223762 1047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:41.224409 1047 catalog_manager.cc:1259] Loaded cluster ID: 6751290c378c4d57a639a5008cccb1ff
I20260924 20:22:41.224470 1047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:41.225513 1047 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 20:22:41.228392 1047 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 20:22:41.239228 1047 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:41.240108 1047 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Loaded TSK: 0
I20260924 20:22:41.241127 1047 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:22:41.306699 1041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:41.307003 1041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:41.307098 1041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:41.311164 1041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:41.311237 1041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:41.311329 1041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:41.316697 1041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:32847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=46839
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:41.317747 1041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:41.318881 1041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:41.329887 1066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:41.329875 1069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:41.329875 1067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.330899 1041 server_base.cc:1034] running on GCE node
I20260924 20:22:41.331344 1041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:41.332006 1041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:41.333201 1041 hybrid_clock.cc:648] HybridClock initialized: now 1790281361333175 us; error 42 us; skew 500 ppm
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:41.336292 1041 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:41.337250 1041 webserver.cc:460] Webserver started at http://127.18.172.65:46839/ using document root <none> and password file <none>
I20260924 20:22:41.337821 1041 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:41.337894 1041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:41.341863 1041 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:22:41.344201 1076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.345515 1041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:41.345657 1041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "1b2912cc83d344659f219f20e30df1f8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "377609370b464631d7fde29d514d0f61"
server_key_iv: "ba785115e3b5041b1708299f099eb2b2"
server_key_version: "encryptionkey@0"
I20260924 20:22:41.346246 1041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:41.373006 1041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:41.376288 1041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:41.376509 1041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:41.377126 1041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:41.378324 1083 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:22:41.382531 1041 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:22:41.382601 1041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260924 20:22:41.382674 1041 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:22:41.384454 1041 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:22:41.384512 1041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 20:22:41.384778 1083 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Bootstrap starting.
I20260924 20:22:41.395392 1041 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:32847
I20260924 20:22:41.395409 1190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:32847 every 8 connection(s)
I20260924 20:22:41.396726 1041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:41.405002 1083 log.cc:824] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:41.406010 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1041
I20260924 20:22:41.406548 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:34601
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=45865
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:41.413910 1083 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:22:41.414391 1083 tablet_bootstrap.cc:492] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Bootstrap complete.
I20260924 20:22:41.414716 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.398892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:49295 (local address 127.18.172.126:45629)
0924 20:22:41.399299 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.399323 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.399659 (+ 336us) server_negotiation.cc:407] Connection header received
0924 20:22:41.400769 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.400779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.401072 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.401393 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:41.402522 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.403550 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.404686 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.405217 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.405403 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.408452 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:41.408461 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:41.408477 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:41.408514 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:41.410657 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.411412 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.411418 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.411423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.411577 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.412024 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.412027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.412029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.412462 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:41.412598 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.413344 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.413643 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":152,"threads_started":1}
I20260924 20:22:41.414747 1194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.398870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:49295)
0924 20:22:41.399480 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.399523 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.400342 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.401449 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.401456 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.401876 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:41.402336 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.402350 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.403715 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.403718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.404557 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.404567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.405362 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.405366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.405572 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.406305 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:41.406332 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:41.408229 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:41.410795 (+ 2566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.410805 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.410821 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.411260 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.411696 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.411701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.411706 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.411897 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.412596 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:41.412602 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:41.412948 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.413181 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.413568 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":181,"threads_started":1}
I20260924 20:22:41.415871 1083 ts_tablet_manager.cc:1397] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.011s
I20260924 20:22:41.417697 1191 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:41.417937 1191 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:41.418577 1191 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:41.418619 1083 raft_consensus.cc:348] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:41.419217 1083 raft_consensus.cc:729] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2912cc83d344659f219f20e30df1f8, State: Initialized, Role: FOLLOWER
I20260924 20:22:41.419703 1083 consensus_queue.cc:260] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } } peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } }
I20260924 20:22:41.421831 1083 ts_tablet_manager.cc:1428] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:22:41.422456 1003 ts_manager.cc:194] Registered new tserver with Master: 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847)
I20260924 20:22:41.426447 1003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:49295
I20260924 20:22:41.429087 1191 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:22:41.437237 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.425778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:32847 (local address 127.0.0.1:51156)
0924 20:22:41.426487 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.426536 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.426767 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.427313 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.427318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.427772 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:41.427904 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.427913 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.429217 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.429221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.430232 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.430244 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.431043 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.431048 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.431189 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.431756 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:41.431769 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:41.433254 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:41.435419 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.435430 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.435436 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.435673 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.435987 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.435990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.435992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.436085 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.436560 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:41.436566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:41.436694 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.436832 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.437034 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":105,"threads_started":1}
I20260924 20:22:41.437237 1199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.425829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.18.172.65:32847)
0924 20:22:41.426227 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.426238 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.426663 (+ 425us) server_negotiation.cc:407] Connection header received
0924 20:22:41.426919 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.426932 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.427145 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.427300 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:41.428050 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.429092 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.430369 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.430875 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.430993 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.433420 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:41.433439 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:41.433454 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:41.433483 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:41.435304 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.435803 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.435808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.435809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.435884 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.436177 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.436182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.436184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.436442 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:41.436596 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.436922 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.437080 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":305,"thread_start_us":97,"threads_started":1}
I20260924 20:22:41.438642 1125 tablet_service.cc:1514] Processing DeleteTablet for tablet d34df94451464229a286414fa4f2e1bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:40 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51156
I20260924 20:22:41.439556 1200 tablet_replica.cc:330] stopping tablet replica
I20260924 20:22:41.439877 1200 raft_consensus.cc:2229] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:41.440145 1200 raft_consensus.cc:2258] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:41.441239 1200 ts_tablet_manager.cc:1905] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:41.445752 1200 ts_tablet_manager.cc:1918] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:41.445855 1200 log.cc:1196] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/d34df94451464229a286414fa4f2e1bd
I20260924 20:22:41.446136 1200 ts_tablet_manager.cc:1939] T d34df94451464229a286414fa4f2e1bd P 1b2912cc83d344659f219f20e30df1f8: Deleting consensus metadata
I20260924 20:22:41.448211 987 catalog_manager.cc:4841] TS 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847): tablet d34df94451464229a286414fa4f2e1bd (table test-table [id=1a678cd6cd4b4d6a8ac23803da63f204]) successfully deleted
W20260924 20:22:41.527729 1196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:41.527961 1196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:41.528002 1196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:41.531913 1196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:41.531966 1196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:41.532022 1196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:22:41.537334 1196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:34601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=45865
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:41.538327 1196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:41.539335 1196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:41.549055 1208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:41.549070 1206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.549154 1196 server_base.cc:1034] running on GCE node
W20260924 20:22:41.549095 1205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.549876 1196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:41.550668 1196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:41.551860 1196 hybrid_clock.cc:648] HybridClock initialized: now 1790281361551797 us; error 74 us; skew 500 ppm
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:41.554659 1196 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:22:41.555565 1196 webserver.cc:460] Webserver started at http://127.18.172.66:45865/ using document root <none> and password file <none>
I20260924 20:22:41.556213 1196 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:41.556313 1196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:41.560221 1196 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260924 20:22:41.562520 1215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.563979 1196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:41.564113 1196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a3a8a78c658a493ebc3d524249cfc30d"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "f0614a34bf29ab3c8fb8e42bf7b1ea1f"
server_key_iv: "6356e003cf6dd0a77520b1bbf5958785"
server_key_version: "encryptionkey@0"
I20260924 20:22:41.564659 1196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:41.591975 1196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:41.595234 1196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:41.595440 1196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:41.596285 1196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:41.597479 1222 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 20:22:41.601313 1222 ts_tablet_manager.cc:1852] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 20:22:41.601707 1222 ts_tablet_manager.cc:1905] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:41.605589 1222 ts_tablet_manager.cc:1918] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:41.605691 1222 log.cc:1196] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/d34df94451464229a286414fa4f2e1bd
I20260924 20:22:41.606031 1222 ts_tablet_manager.cc:1939] T d34df94451464229a286414fa4f2e1bd P a3a8a78c658a493ebc3d524249cfc30d: Deleting consensus metadata
I20260924 20:22:41.606371 1196 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 20:22:41.606451 1196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260924 20:22:41.606551 1196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:41.606606 1196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.615954 1196 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:34601
I20260924 20:22:41.615990 1329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:34601 every 8 connection(s)
I20260924 20:22:41.616869 1196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:22:41.624343 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1196
I20260924 20:22:41.624918 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:42773
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=33949
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:41.630985 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.618760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:36961 (local address 127.18.172.126:45629)
0924 20:22:41.618943 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.618947 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.619966 (+ 1019us) server_negotiation.cc:407] Connection header received
0924 20:22:41.620894 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.620896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.620947 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.621023 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:41.622201 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.622717 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.623849 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.624067 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.624147 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.627025 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:41.627030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:41.627036 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:41.627068 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:41.628629 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.629304 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.629307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.629308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.629379 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.629799 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.629802 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.629803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.629983 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:41.630061 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.630747 (+ 686us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.630861 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 20:22:41.632391 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.619259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:36961)
0924 20:22:41.619808 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.619851 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.620712 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.621184 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.621190 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.621586 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:41.622069 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.622082 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.622825 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.622827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.623741 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.623750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.624224 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.624228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.624421 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.625050 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:41.625077 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:41.626821 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:41.628755 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.628764 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.628782 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.629201 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.629475 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.629478 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.629480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.629689 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.630095 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:41.630104 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:41.630467 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.630714 (+ 247us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.631144 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":106,"threads_started":1}
I20260924 20:22:41.633709 1330 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:41.633975 1330 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:41.634479 1330 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:41.635885 1003 ts_manager.cc:194] Registered new tserver with Master: a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601)
I20260924 20:22:41.636576 1003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:36961
W20260924 20:22:41.737960 1333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:41.738252 1333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:41.738324 1333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:41.742439 1333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:41.742538 1333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:41.742625 1333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:22:41.748725 1333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:42773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=33949
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:41.749872 1333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:41.750955 1333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:41.760589 1341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.760633 1333 server_base.cc:1034] running on GCE node
W20260924 20:22:41.760591 1338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:41.760589 1339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:41.761199 1333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:41.761811 1333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:41.763054 1333 hybrid_clock.cc:648] HybridClock initialized: now 1790281361762957 us; error 53 us; skew 500 ppm
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:41.766119 1333 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:22:41.767242 1333 webserver.cc:460] Webserver started at http://127.18.172.67:33949/ using document root <none> and password file <none>
I20260924 20:22:41.767956 1333 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:41.768064 1333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:41.772416 1333 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:22:41.774545 1348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.775923 1333 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:22:41.776074 1333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "21f71bb592894c649988350dd5af12a8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "3d9e0936fc6912cb8b24a3a525c0c4a3"
server_key_iv: "e6d27d21f8fe3c45fcd0cda10a3f98f6"
server_key_version: "encryptionkey@0"
I20260924 20:22:41.776670 1333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:41.796933 1333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:41.800534 1333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:41.800751 1333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:41.801381 1333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:41.802259 1333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:41.802313 1333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.802356 1333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:41.802380 1333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:41.811897 1333 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42773
I20260924 20:22:41.811962 1461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42773 every 8 connection(s)
I20260924 20:22:41.812999 1333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:22:41.822096 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1333
Sep 24 20:22:41 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281361, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:41.828362 1193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.814846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:32933 (local address 127.18.172.126:45629)
0924 20:22:41.814991 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.814995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.815957 (+ 962us) server_negotiation.cc:407] Connection header received
0924 20:22:41.817404 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.817407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.817470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.817602 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:41.818920 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.819514 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.820939 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.821155 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.821230 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.824270 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:41.824273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:41.824279 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:41.824315 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:41.825790 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.826466 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.826469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.826470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.826563 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.827034 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.827039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.827041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.827305 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:41.827446 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.828005 (+ 559us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.828153 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:22:41.829208 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.815362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:32933)
0924 20:22:41.815805 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.815848 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.817172 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.817832 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.817840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.818209 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:41.818779 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.818795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.819655 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.819658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.820819 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.820831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.821311 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.821315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.821479 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.822114 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:41.822139 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:41.824101 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:41.825908 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.825914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.825929 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.826229 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.826714 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:41.826720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:41.826725 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:41.826873 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:41.827455 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:41.827462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:41.827721 (+ 259us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.827889 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.828156 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260924 20:22:41.830572 1462 heartbeater.cc:347] Connected to a master server at 127.18.172.126:45629
I20260924 20:22:41.830858 1462 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:41.831513 1462 heartbeater.cc:511] Master 127.18.172.126:45629 requested a full tablet report, sending...
I20260924 20:22:41.832414 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.822908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.18.172.126:45629)
0924 20:22:41.823182 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.823185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.823327 (+ 142us) server_negotiation.cc:407] Connection header received
0924 20:22:41.823473 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.823475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.823513 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.823585 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:41.824381 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.825154 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.826258 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.826535 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.826617 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.828628 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:41.828631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:41.828633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:41.828662 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:41.830545 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.831068 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.831072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.831074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.831175 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:41.831516 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:41.831519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:41.831522 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:41.831732 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:41.831986 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.832148 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.832251 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":81,"threads_started":1}
I20260924 20:22:41.832738 1003 ts_manager.cc:194] Registered new tserver with Master: 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
I20260924 20:22:41.833457 1003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:32933
I20260924 20:22:41.834215 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:22:41.838513 1466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.834857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.18.172.126:45629)
0924 20:22:41.834964 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.834968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.835174 (+ 206us) server_negotiation.cc:407] Connection header received
0924 20:22:41.835281 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.835284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.835395 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.835499 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:22:41.835759 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.836260 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.837208 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.837421 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.837498 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.837816 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:22:41.838163 (+ 347us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:22:41.838249 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.838375 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.838423 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:22:41.839345 1002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:22:41.842471 1002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:22:41.850626 1125 tablet_service.cc:1467] Processing CreateTablet for tablet 29b118fde6684e61b6f60c95f7319493 (DEFAULT_TABLE table=test-table [id=65ee7c4e28384bc4ab76068297418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:41.851410 1125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b118fde6684e61b6f60c95f7319493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:41.854295 1083 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Bootstrap starting.
I20260924 20:22:41.855239 1083 tablet_bootstrap.cc:654] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:41.855392 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.849465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34601 (local address 127.0.0.1:36612)
0924 20:22:41.849704 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.849719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.849800 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.850772 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.850775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.850794 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:41.850927 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.850934 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.852331 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.852335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.853520 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.853531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.854935 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.854939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.855066 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.855125 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.855179 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.855258 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1}
I20260924 20:22:41.856043 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.849677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.18.172.66:34601)
0924 20:22:41.850244 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.850258 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.850284 (+ 26us) server_negotiation.cc:407] Connection header received
0924 20:22:41.850363 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.850368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.850598 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.851207 (+ 609us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:41.851385 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.852196 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.853692 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.854797 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.855570 (+ 773us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.855718 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.855772 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.855866 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":150,"threads_started":1}
I20260924 20:22:41.856197 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.849660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42773 (local address 127.0.0.1:60424)
0924 20:22:41.849978 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:41.849993 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:41.850080 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:41.850885 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:41.850887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:41.850901 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:41.852399 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.852405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.853395 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.853400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.854513 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:41.854523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.855781 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.855784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:41.855877 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.855899 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:22:41.855982 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:22:41.856047 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1392,"thread_start_us":120,"threads_started":1}
I20260924 20:22:41.856308 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:41.850084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.18.172.67:42773)
0924 20:22:41.850399 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:41.850409 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:41.850427 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:22:41.850484 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:41.850489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:41.850627 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:41.850952 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:41.852525 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.853255 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.854650 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:41.855480 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:41.855577 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:41.855718 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:41.856035 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 20:22:41.856147 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":105,"threads_started":1}
I20260924 20:22:41.856997 1083 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: No bootstrap required, opened a new log
I20260924 20:22:41.857085 1083 ts_tablet_manager.cc:1397] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:22:41.857540 1083 raft_consensus.cc:348] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.857697 1083 raft_consensus.cc:374] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:41.857750 1083 raft_consensus.cc:729] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2912cc83d344659f219f20e30df1f8, State: Initialized, Role: FOLLOWER
I20260924 20:22:41.857879 1083 consensus_queue.cc:260] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.858135 1083 ts_tablet_manager.cc:1428] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 20:22:41.858909 1264 tablet_service.cc:1467] Processing CreateTablet for tablet 29b118fde6684e61b6f60c95f7319493 (DEFAULT_TABLE table=test-table [id=65ee7c4e28384bc4ab76068297418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:41.859442 1396 tablet_service.cc:1467] Processing CreateTablet for tablet 29b118fde6684e61b6f60c95f7319493 (DEFAULT_TABLE table=test-table [id=65ee7c4e28384bc4ab76068297418e37]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:22:41.859987 1264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b118fde6684e61b6f60c95f7319493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:41.860497 1396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29b118fde6684e61b6f60c95f7319493. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:41.864583 1222 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Bootstrap starting.
I20260924 20:22:41.866420 1222 tablet_bootstrap.cc:654] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:41.867265 1222 log.cc:824] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:41.867561 1475 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Bootstrap starting.
I20260924 20:22:41.869226 1222 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: No bootstrap required, opened a new log
I20260924 20:22:41.869413 1222 ts_tablet_manager.cc:1397] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:22:41.869846 1475 tablet_bootstrap.cc:654] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:41.870841 1475 log.cc:824] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:41.872870 1475 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: No bootstrap required, opened a new log
I20260924 20:22:41.872773 1222 raft_consensus.cc:348] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.872985 1222 raft_consensus.cc:374] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:41.873054 1222 raft_consensus.cc:729] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3a8a78c658a493ebc3d524249cfc30d, State: Initialized, Role: FOLLOWER
I20260924 20:22:41.873097 1475 ts_tablet_manager.cc:1397] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:22:41.873535 1222 consensus_queue.cc:260] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.874291 1330 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
I20260924 20:22:41.874571 1222 ts_tablet_manager.cc:1428] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260924 20:22:41.876407 1475 raft_consensus.cc:348] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.876634 1475 raft_consensus.cc:374] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:41.876690 1475 raft_consensus.cc:729] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f71bb592894c649988350dd5af12a8, State: Initialized, Role: FOLLOWER
I20260924 20:22:41.877130 1475 consensus_queue.cc:260] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:41.877733 1462 heartbeater.cc:503] Master 127.18.172.126:45629 was elected leader, sending a full tablet report...
I20260924 20:22:41.878098 1475 ts_tablet_manager.cc:1428] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 20:22:41.897902 1192 tablet.cc:2369] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:22:42.064016 1463 tablet.cc:2369] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:22:42.118041 1331 tablet.cc:2369] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:22:42.216118 1197 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:42.216347 1197 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:42.217446 1197 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
I20260924 20:22:42.221844 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.217932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:52731 (local address 127.18.172.67:42773)
0924 20:22:42.218064 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:42.218068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:42.218107 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:22:42.218314 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:42.218318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:42.218368 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:42.218459 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:42.218877 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.219558 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.220820 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.221248 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.221362 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:42.221420 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:42.221647 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 20:22:42.221714 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:22:42.221873 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.217771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:40307 (local address 127.18.172.66:34601)
0924 20:22:42.217903 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:42.217909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:42.218116 (+ 207us) server_negotiation.cc:407] Connection header received
0924 20:22:42.218314 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:42.218318 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:42.218369 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:42.218451 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:22:42.218919 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.219561 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.220889 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.221268 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.221404 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:42.221456 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:42.221682 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 20:22:42.221762 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:22:42.221881 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.217691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42773 (local address 127.18.172.65:52731)
0924 20:22:42.218006 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.218023 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:42.218171 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:42.218481 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:42.218484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:42.218510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:42.218733 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:42.218744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.219671 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.219677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:42.220701 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:42.220710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.221382 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.221386 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:42.221490 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:42.221540 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 20:22:42.221601 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 20:22:42.221695 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":144,"spinlock_wait_cycles":18048,"thread_start_us":86,"threads_started":1}
I20260924 20:22:42.221885 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.217687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34601 (local address 127.18.172.65:40307)
0924 20:22:42.218007 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.218028 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:42.218171 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:42.218481 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:42.218484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:42.218510 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:22:42.218753 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:42.218764 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.219719 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.219721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:42.220736 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:42.220745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.221419 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.221422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:42.221512 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:42.221550 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:22:42.221644 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:22:42.221714 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":168,"spinlock_wait_cycles":53760,"thread_start_us":74,"threads_started":1}
I20260924 20:22:42.222656 1416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b118fde6684e61b6f60c95f7319493" candidate_uuid: "1b2912cc83d344659f219f20e30df1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f71bb592894c649988350dd5af12a8" is_pre_election: true
I20260924 20:22:42.222826 1284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b118fde6684e61b6f60c95f7319493" candidate_uuid: "1b2912cc83d344659f219f20e30df1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a8a78c658a493ebc3d524249cfc30d" is_pre_election: true
I20260924 20:22:42.223079 1416 raft_consensus.cc:2454] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b2912cc83d344659f219f20e30df1f8 in term 0.
I20260924 20:22:42.223168 1284 raft_consensus.cc:2454] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b2912cc83d344659f219f20e30df1f8 in term 0.
I20260924 20:22:42.223559 1079 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8, 21f71bb592894c649988350dd5af12a8; no voters:
I20260924 20:22:42.223845 1197 raft_consensus.cc:2781] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:22:42.223932 1197 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:22:42.223964 1197 raft_consensus.cc:3037] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:42.225018 1197 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:42.225422 1197 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 1 election: Requested vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
I20260924 20:22:42.225757 1284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b118fde6684e61b6f60c95f7319493" candidate_uuid: "1b2912cc83d344659f219f20e30df1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3a8a78c658a493ebc3d524249cfc30d"
I20260924 20:22:42.225800 1416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29b118fde6684e61b6f60c95f7319493" candidate_uuid: "1b2912cc83d344659f219f20e30df1f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f71bb592894c649988350dd5af12a8"
I20260924 20:22:42.225894 1284 raft_consensus.cc:3037] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:42.225962 1416 raft_consensus.cc:3037] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:42.226948 1416 raft_consensus.cc:2454] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b2912cc83d344659f219f20e30df1f8 in term 1.
I20260924 20:22:42.226966 1284 raft_consensus.cc:2454] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b2912cc83d344659f219f20e30df1f8 in term 1.
I20260924 20:22:42.227321 1080 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8, a3a8a78c658a493ebc3d524249cfc30d; no voters:
I20260924 20:22:42.227547 1197 raft_consensus.cc:2781] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:42.227702 1197 raft_consensus.cc:686] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 LEADER]: Becoming Leader. State: Replica: 1b2912cc83d344659f219f20e30df1f8, State: Running, Role: LEADER
I20260924 20:22:42.227964 1197 consensus_queue.cc:237] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:42.230741 1002 catalog_manager.cc:5495] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "1b2912cc83d344659f219f20e30df1f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } health_report { overall_health: HEALTHY } } }
I20260924 20:22:42.313082 1485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.309370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51172 (local address 127.18.172.65:32847)
0924 20:22:42.309667 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:42.309673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:42.309687 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:22:42.309760 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:42.309762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:42.309814 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:42.309910 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:22:42.310126 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.310723 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.311621 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:42.311846 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:42.311921 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:42.312150 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:22:42.312575 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:22:42.312658 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:42.312820 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 20:22:42.312908 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260924 20:22:42.317193 1416 raft_consensus.cc:1261] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Refusing update from remote peer 1b2912cc83d344659f219f20e30df1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:42.317358 1284 raft_consensus.cc:1261] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Refusing update from remote peer 1b2912cc83d344659f219f20e30df1f8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:22:42.318190 1482 consensus_queue.cc:1035] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:42.318423 1197 consensus_queue.cc:1035] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:22:42.326722 1486 mvcc.cc:205] Tried to move back new op lower bound from 7332992460043776000 to 7332992459688370176. Current Snapshot: MvccSnapshot[applied={T|T < 7332992460043776000}]
I20260924 20:22:42.330446 1487 mvcc.cc:205] Tried to move back new op lower bound from 7332992460043776000 to 7332992459688370176. Current Snapshot: MvccSnapshot[applied={T|T < 7332992460043776000}]
I20260924 20:22:42.330948 1489 mvcc.cc:205] Tried to move back new op lower bound from 7332992460043776000 to 7332992459688370176. Current Snapshot: MvccSnapshot[applied={T|T < 7332992460043776000}]
I20260924 20:22:42.337234 1003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:42.337432 1003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:22:42.339979 1003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Sending DeleteTablet for 3 replicas of tablet 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:42.340727 1264 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b118fde6684e61b6f60c95f7319493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:42 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:36612
I20260924 20:22:42.340727 1396 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b118fde6684e61b6f60c95f7319493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:42 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:60424
I20260924 20:22:42.341275 1496 tablet_replica.cc:330] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: stopping tablet replica
I20260924 20:22:42.341435 1495 tablet_replica.cc:330] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: stopping tablet replica
I20260924 20:22:42.341428 1125 tablet_service.cc:1514] Processing DeleteTablet for tablet 29b118fde6684e61b6f60c95f7319493 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:22:42 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:51156
I20260924 20:22:42.341778 1495 raft_consensus.cc:2229] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:42.341840 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1041
I20260924 20:22:42.341933 1496 raft_consensus.cc:2229] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:22:42.342130 1495 raft_consensus.cc:2258] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:42.342456 1496 raft_consensus.cc:2258] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:22:42.343845 1495 ts_tablet_manager.cc:1905] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:42.344247 1496 ts_tablet_manager.cc:1905] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:22:42.346164 1495 ts_tablet_manager.cc:1918] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:42.346290 1495 log.cc:1196] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/29b118fde6684e61b6f60c95f7319493
I20260924 20:22:42.346673 1495 ts_tablet_manager.cc:1939] T 29b118fde6684e61b6f60c95f7319493 P 21f71bb592894c649988350dd5af12a8: Deleting consensus metadata
I20260924 20:22:42.348034 1496 ts_tablet_manager.cc:1918] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:22:42.348034 989 catalog_manager.cc:4841] TS 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): tablet 29b118fde6684e61b6f60c95f7319493 (table test-table [id=65ee7c4e28384bc4ab76068297418e37]) successfully deleted
I20260924 20:22:42.348129 1496 log.cc:1196] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/29b118fde6684e61b6f60c95f7319493
I20260924 20:22:42.348431 1496 ts_tablet_manager.cc:1939] T 29b118fde6684e61b6f60c95f7319493 P a3a8a78c658a493ebc3d524249cfc30d: Deleting consensus metadata
I20260924 20:22:42.349710 990 catalog_manager.cc:4841] TS a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): tablet 29b118fde6684e61b6f60c95f7319493 (table test-table [id=65ee7c4e28384bc4ab76068297418e37]) successfully deleted
W20260924 20:22:42.350373 987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:32847 (error 108)
I20260924 20:22:42.350646 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1196
I20260924 20:22:42.351464 1471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.351150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:32847 (local address 127.0.0.1:51174)
0924 20:22:42.351335 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.351390 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:32847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260924 20:22:42.351601 987 catalog_manager.cc:4584] TS 1b2912cc83d344659f219f20e30df1f8 (127.18.172.65:32847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29b118fde6684e61b6f60c95f7319493: Network error: Client connection negotiation failed: client connection to 127.18.172.65:32847: connect: Connection refused (error 111)
I20260924 20:22:42.358093 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1333
I20260924 20:22:42.364157 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 970
I20260924 20:22:42.371477 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--webserver_interface=127.18.172.126
--webserver_port=46289
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:45629
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:42.480183 1499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:42.480449 1499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:42.480497 1499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:42.484473 1499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:42.484531 1499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:42.484550 1499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:42.484567 1499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:42.484582 1499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:42.489884 1499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:45629
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:45629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=46289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:42.490911 1499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:42.491967 1499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:42.501492 1504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:42.501569 1499 server_base.cc:1034] running on GCE node
W20260924 20:22:42.501502 1507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:42.501488 1505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:42.502326 1499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:42.503634 1499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:42.504818 1499 hybrid_clock.cc:648] HybridClock initialized: now 1790281362504781 us; error 52 us; skew 500 ppm
Sep 24 20:22:42 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281362, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:42.507786 1499 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:42.508656 1499 webserver.cc:460] Webserver started at http://127.18.172.126:46289/ using document root <none> and password file <none>
I20260924 20:22:42.509169 1499 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:42.509230 1499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:42.513087 1499 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:22:42.515239 1514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:42.516520 1499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:22:42.516691 1499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "37f81d4e27c74b5f98d4df2f9c175d52"
format_stamp: "Formatted at 2026-09-24 20:22:39 on dist-test-slave-m1qq"
server_key: "c2214fde8f3e6c7ecfa777d3d4584de3"
server_key_iv: "b2c2e3b9772b650022136486001e5da9"
server_key_version: "encryptionkey@0"
I20260924 20:22:42.517282 1499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:42.528321 1499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:42.531198 1499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:42.531404 1499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:42.538513 1499 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:45629
I20260924 20:22:42.538825 1566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:45629 every 8 connection(s)
I20260924 20:22:42.539723 1499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:42.544040 1567 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:22:42.545696 1567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap starting.
I20260924 20:22:42.547744 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1499
I20260924 20:22:42.548220 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:32847
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=46839
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:42.563576 1567 log.cc:824] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:42.575496 1567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:22:42.575910 1567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52: Bootstrap complete.
I20260924 20:22:42.580089 1567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:42.580521 1567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Initialized, Role: FOLLOWER
I20260924 20:22:42.581070 1567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:42.581231 1567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:42.581324 1567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:42.581442 1567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:22:42.583468 1567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:42.583806 1567 leader_election.cc:302] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f81d4e27c74b5f98d4df2f9c175d52; no voters:
I20260924 20:22:42.584141 1567 leader_election.cc:288] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:22:42.584283 1571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:22:42.584555 1571 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [term 3 LEADER]: Becoming Leader. State: Replica: 37f81d4e27c74b5f98d4df2f9c175d52, State: Running, Role: LEADER
I20260924 20:22:42.584873 1571 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } }
I20260924 20:22:42.585376 1567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:42.586365 1573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f81d4e27c74b5f98d4df2f9c175d52. Latest consensus state: current_term: 3 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:42.586391 1572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f81d4e27c74b5f98d4df2f9c175d52" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 45629 } } }
I20260924 20:22:42.586495 1572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:42.586496 1573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f81d4e27c74b5f98d4df2f9c175d52 [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:42.586875 1576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:42.592057 1576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:42.592756 1576 catalog_manager.cc:1259] Loaded cluster ID: 6751290c378c4d57a639a5008cccb1ff
I20260924 20:22:42.592815 1576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 20:22:42.593809 1576 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f810c41d9ad google::LogMessage::Fail() at ??:0
@ 0x7f810c42088a google::LogMessage::SendToLog() at ??:0
@ 0x7f810c41d4de google::LogMessage::Flush() at ??:0
@ 0x7f810c41ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8116d60fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8116d61d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8116d5e579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8116da2640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f810e66f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f810e66fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f810e6713f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f810e660512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f810fca46db start_thread at ??:0
@ 0x7f810af1b61f clone at ??:0
W20260924 20:22:42.682333 1569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:42.682633 1569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:42.682708 1569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:42.687071 1569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:42.687155 1569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:42.687281 1569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:42.692909 1569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:32847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=46839
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:42.694043 1569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:42.695125 1569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:42.705174 1608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:42.705217 1611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:42.705180 1609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:42.705657 1569 server_base.cc:1034] running on GCE node
I20260924 20:22:42.706111 1569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:42.706919 1569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:42.708132 1569 hybrid_clock.cc:648] HybridClock initialized: now 1790281362708099 us; error 32 us; skew 500 ppm
Sep 24 20:22:42 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281362, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:42.711622 1569 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:42.712803 1569 webserver.cc:460] Webserver started at http://127.18.172.65:46839/ using document root <none> and password file <none>
I20260924 20:22:42.713605 1569 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:42.713696 1569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:42.718802 1569 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:22:42.721601 1618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:42.723263 1569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:42.723414 1569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "1b2912cc83d344659f219f20e30df1f8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "377609370b464631d7fde29d514d0f61"
server_key_iv: "ba785115e3b5041b1708299f099eb2b2"
server_key_version: "encryptionkey@0"
I20260924 20:22:42.724164 1569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:42.740765 1569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:42.744769 1569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:42.744974 1569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:42.745716 1569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:42.747151 1625 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:22:42.751405 1569 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:22:42.751482 1569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260924 20:22:42.751550 1569 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:22:42.753705 1569 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:22:42.753762 1569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:22:42.754132 1625 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Bootstrap starting.
I20260924 20:22:42.764592 1569 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:32847
I20260924 20:22:42.764742 1732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:32847 every 8 connection(s)
I20260924 20:22:42.765774 1569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:42.775370 1625 log.cc:824] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:42.775478 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1569
I20260924 20:22:42.775971 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:34601
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=45865
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:22:42.782877 1625 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:22:42.783300 1625 tablet_bootstrap.cc:492] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Bootstrap complete.
I20260924 20:22:42.784399 1625 ts_tablet_manager.cc:1397] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.007s
I20260924 20:22:42.786818 1625 raft_consensus.cc:348] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:42.787302 1625 raft_consensus.cc:729] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2912cc83d344659f219f20e30df1f8, State: Initialized, Role: FOLLOWER
I20260924 20:22:42.787796 1625 consensus_queue.cc:260] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:42.788774 1625 ts_tablet_manager.cc:1428] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260924 20:22:42.849709 1735 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:22:42.768157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:37421)
0924 20:22:42.769354 (+ 1197us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.769413 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:42.770436 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:42.848956 (+ 78520us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":966,"thread_start_us":148,"threads_started":1}
W20260924 20:22:42.850348 1733 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:45629 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:45629: Client connection negotiation failed: client connection to 127.18.172.126:45629: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:22:42.851126 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.850840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:44605)
0924 20:22:42.850965 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.851063 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:22:42.851811 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.851617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:36091)
0924 20:22:42.851724 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:42.851766 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 20:22:42.852054 1733 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:22:42.925741 1737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:42.926016 1737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:42.926062 1737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:42.930706 1737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:42.930773 1737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:42.930871 1737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:22:42.936702 1737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:34601
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=45865
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:42.937791 1737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:42.938819 1737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:42.949987 1743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:42.949955 1744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:42.950169 1737 server_base.cc:1034] running on GCE node
W20260924 20:22:42.949962 1746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:42.950551 1737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:42.951299 1737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:42.952489 1737 hybrid_clock.cc:648] HybridClock initialized: now 1790281362952474 us; error 26 us; skew 500 ppm
Sep 24 20:22:42 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281362, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:42.955265 1737 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:22:42.956095 1737 webserver.cc:460] Webserver started at http://127.18.172.66:45865/ using document root <none> and password file <none>
I20260924 20:22:42.956666 1737 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:42.956761 1737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:42.960887 1737 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:22:42.963785 1753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:42.965154 1737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:42.965286 1737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a3a8a78c658a493ebc3d524249cfc30d"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "f0614a34bf29ab3c8fb8e42bf7b1ea1f"
server_key_iv: "6356e003cf6dd0a77520b1bbf5958785"
server_key_version: "encryptionkey@0"
I20260924 20:22:42.965855 1737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:42.982232 1737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:42.985231 1737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:42.985435 1737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:42.986016 1737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:42.986837 1737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:42.986910 1737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:42.986982 1737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:42.987064 1737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:42.996104 1737 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:34601
I20260924 20:22:42.996124 1866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:34601 every 8 connection(s)
I20260924 20:22:42.997138 1737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:22:43.001253 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:42.999492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:52531)
0924 20:22:42.999950 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.000074 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
W20260924 20:22:43.001811 1867 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:45629 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:45629: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
I20260924 20:22:43.002426 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.002207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:47201)
0924 20:22:43.002320 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.002370 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 20:22:43.003295 1869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.003104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:52887)
0924 20:22:43.003201 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.003244 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 20:22:43.003474 1867 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:22:43.005941 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1737
I20260924 20:22:43.006346 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:42773
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:45629
--webserver_port=33949
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:43.125510 1870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:43.125765 1870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:43.125835 1870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:43.129918 1870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:43.129981 1870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:43.130065 1870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:22:43.135651 1870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:42773
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=33949
--tserver_master_addrs=127.18.172.126:45629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:43.136744 1870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:43.137773 1870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:43.147557 1878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:43.147558 1876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:43.147564 1875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:43.147750 1870 server_base.cc:1034] running on GCE node
I20260924 20:22:43.148324 1870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:43.148963 1870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:43.150152 1870 hybrid_clock.cc:648] HybridClock initialized: now 1790281363150135 us; error 39 us; skew 500 ppm
Sep 24 20:22:43 dist-test-slave-m1qq krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281363, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:43.152931 1870 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:22:43.153831 1870 webserver.cc:460] Webserver started at http://127.18.172.67:33949/ using document root <none> and password file <none>
I20260924 20:22:43.154383 1870 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:43.154476 1870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:43.158360 1870 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260924 20:22:43.160450 1885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:43.161721 1870 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:22:43.161854 1870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "21f71bb592894c649988350dd5af12a8"
format_stamp: "Formatted at 2026-09-24 20:22:40 on dist-test-slave-m1qq"
server_key: "3d9e0936fc6912cb8b24a3a525c0c4a3"
server_key_iv: "e6d27d21f8fe3c45fcd0cda10a3f98f6"
server_key_version: "encryptionkey@0"
I20260924 20:22:43.162395 1870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:43.172685 1870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:43.175547 1870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:43.175751 1870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:43.176306 1870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:43.177107 1870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:43.177179 1870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:43.177238 1870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:43.177291 1870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:43.186741 1870 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42773
I20260924 20:22:43.186765 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42773 every 8 connection(s)
I20260924 20:22:43.187767 1870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:22:43.192152 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.189912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:57939)
0924 20:22:43.190432 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.190593 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":106,"threads_started":1}
I20260924 20:22:43.192811 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 1870
W20260924 20:22:43.194231 1999 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:45629 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:45629: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
I20260924 20:22:43.194875 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.194640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:57205)
0924 20:22:43.194774 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.194817 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 20:22:43.195694 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.195282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:40223)
0924 20:22:43.195525 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.195605 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
W20260924 20:22:43.195834 1999 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:22:43.853228 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:43.852779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:33221)
0924 20:22:43.853052 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:43.853111 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 20:22:44.004463 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.004038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:59367)
0924 20:22:44.004295 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:44.004362 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260924 20:22:44.196853 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.196386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:36785)
0924 20:22:44.196691 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:44.196753 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":82,"threads_started":1}
I20260924 20:22:44.286293 2007 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:44.286527 2007 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:44.288075 2007 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
Sep 24 20:22:44 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281362, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.67@KRBTEST.COM
Sep 24 20:22:44 dist-test-slave-m1qq krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281362, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.66@KRBTEST.COM
I20260924 20:22:44.302539 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.288321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42773 (local address 127.18.172.65:50211)
0924 20:22:44.288777 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:44.288791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:44.288867 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:44.290371 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:44.290377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:44.291129 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:44.291447 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:44.291491 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.293350 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.293353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:44.294247 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:44.294256 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.295112 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.295115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:44.295214 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:44.295901 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:44.295913 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:44.297511 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:44.299898 (+ 2387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:44.299904 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:44.299918 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:44.300196 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:44.300583 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300586 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:44.300587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:44.300733 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:44.301542 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:44.301548 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:44.301842 (+ 294us) client_negotiation.cc:769] Sending connection context
0924 20:22:44.302093 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 20:22:44.302342 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":214,"spinlock_wait_cycles":8192,"thread_start_us":78,"threads_started":1}
I20260924 20:22:44.302636 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.288315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34601 (local address 127.18.172.65:38341)
0924 20:22:44.288475 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:44.288493 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:44.288612 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:44.290369 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:44.290375 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:44.290874 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:44.291447 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:44.291467 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.293337 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.293340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:44.294250 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:44.294261 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.294980 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.294986 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:44.295210 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:44.295861 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:44.295878 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:44.297909 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:44.300203 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:44.300208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:44.300409 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:44.300767 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:44.300778 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:44.300865 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:44.301541 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:44.301615 (+ 74us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:44.301842 (+ 227us) client_negotiation.cc:769] Sending connection context
0924 20:22:44.302058 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 20:22:44.302486 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":110,"spinlock_wait_cycles":6400}
I20260924 20:22:44.303326 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.288432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:38341 (local address 127.18.172.66:34601)
0924 20:22:44.288874 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:44.288899 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:44.288947 (+ 48us) server_negotiation.cc:407] Connection header received
0924 20:22:44.289669 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:44.289675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:44.290019 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:44.290322 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:44.291633 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.293175 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.294383 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.294848 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.294969 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:44.298092 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:44.298097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:44.298107 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:44.298159 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:44.300096 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300516 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:44.300524 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:44.300532 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:44.300672 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300952 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:44.300955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:44.300957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:44.301384 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:44.301523 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:44.302843 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0924 20:22:44.303153 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":72,"threads_started":1}
I20260924 20:22:44.303340 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.288561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:50211 (local address 127.18.172.67:42773)
0924 20:22:44.288908 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:44.288919 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:44.288953 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:22:44.289684 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:44.289690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:44.290019 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:44.290336 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:44.291669 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.293176 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.294382 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:44.294958 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:44.295159 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:44.297766 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:44.297774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:44.297789 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:44.297825 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:44.299778 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300304 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:44.300309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:44.300313 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:44.300433 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:44.300839 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:44.300842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:44.300843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:44.301384 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:44.301523 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:44.302840 (+ 1317us) server_negotiation.cc:299] Negotiation successful
0924 20:22:44.303153 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
W20260924 20:22:44.305446 1621 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
W20260924 20:22:44.305622 1622 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:44.305716 1622 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8; no voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d
I20260924 20:22:44.305989 2007 raft_consensus.cc:2726] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:22:44.854544 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:44.854102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:54493)
0924 20:22:44.854377 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:44.854445 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":92,"threads_started":1}
I20260924 20:22:45.005688 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:45.005269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:35841)
0924 20:22:45.005525 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:45.005596 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":73,"threads_started":1}
I20260924 20:22:45.197974 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:45.197585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:49523)
0924 20:22:45.197837 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:45.197890 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":70,"threads_started":1}
I20260924 20:22:45.855836 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:45.855341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:57941)
0924 20:22:45.855682 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:45.855738 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":70,"threads_started":1}
I20260924 20:22:46.006891 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:46.006465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:48827)
0924 20:22:46.006748 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:46.006800 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":81,"threads_started":1}
I20260924 20:22:46.132704 2016 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:46.132839 2016 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:46.133358 2016 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
W20260924 20:22:46.134199 1622 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
W20260924 20:22:46.134353 1621 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:46.134662 1621 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8; no voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d
I20260924 20:22:46.134858 2016 raft_consensus.cc:2726] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:22:46.199163 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:46.198732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:34809)
0924 20:22:46.198965 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:46.199075 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":68,"threads_started":1}
I20260924 20:22:46.857064 2018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:46.856616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:56349)
0924 20:22:46.856907 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:46.856967 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260924 20:22:47.008258 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:47.007821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:60735)
0924 20:22:47.008100 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:47.008164 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":74,"threads_started":1}
I20260924 20:22:47.200434 2020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:47.199970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:60203)
0924 20:22:47.200281 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:47.200343 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":121,"threads_started":1}
I20260924 20:22:47.858573 2021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:47.858038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:52153)
0924 20:22:47.858420 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:47.858482 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":194,"threads_started":1}
I20260924 20:22:48.009616 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:48.009170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:57839)
0924 20:22:48.009459 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:48.009515 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260924 20:22:48.201738 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:48.201254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:38735)
0924 20:22:48.201563 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:48.201623 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":73,"threads_started":1}
I20260924 20:22:48.859819 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:48.859368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:56959)
0924 20:22:48.859640 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:48.859718 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":100,"threads_started":1}
I20260924 20:22:49.010972 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:49.010497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:44387)
0924 20:22:49.010811 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:49.010874 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":90,"threads_started":1}
I20260924 20:22:49.038951 2026 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:49.039188 2026 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:49.039614 2026 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
W20260924 20:22:49.040396 1621 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
W20260924 20:22:49.040500 1622 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:49.040632 1622 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8; no voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d
I20260924 20:22:49.040799 2026 raft_consensus.cc:2726] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:22:49.203094 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:49.202584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:53065)
0924 20:22:49.202859 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:49.202919 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260924 20:22:49.861007 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:49.860549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:57069)
0924 20:22:49.860844 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:49.860905 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":84,"threads_started":1}
I20260924 20:22:50.012398 2029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:50.011877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:57983)
0924 20:22:50.012237 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:50.012301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":79,"threads_started":1}
I20260924 20:22:50.204511 2030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:50.204014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:56499)
0924 20:22:50.204342 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:50.204407 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
I20260924 20:22:50.862227 2031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:50.861816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:34859)
0924 20:22:50.862072 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:50.862131 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260924 20:22:51.013850 2032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:51.013289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:54467)
0924 20:22:51.013691 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:51.013756 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":80,"threads_started":1}
I20260924 20:22:51.206125 2033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:51.205320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:39007)
0924 20:22:51.205899 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:51.205969 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":101,"threads_started":1}
I20260924 20:22:51.545334 2034 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:51.545511 2034 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:51.546053 2034 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
W20260924 20:22:51.546847 1622 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
W20260924 20:22:51.546972 1621 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:51.547101 1621 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8; no voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d
I20260924 20:22:51.547262 2034 raft_consensus.cc:2726] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:22:51.863498 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:51.863050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:40033)
0924 20:22:51.863337 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:51.863401 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":70,"threads_started":1}
I20260924 20:22:52.015358 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:52.014791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:50323)
0924 20:22:52.015152 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:52.015221 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":81,"threads_started":1}
I20260924 20:22:52.207369 2037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:52.206875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:44951)
0924 20:22:52.207184 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:52.207272 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":93,"threads_started":1}
I20260924 20:22:52.865115 2038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:52.864578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:55389)
0924 20:22:52.864942 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:52.865020 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":89,"threads_started":1}
I20260924 20:22:53.016821 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:53.016290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:52149)
0924 20:22:53.016591 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:53.016666 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":102,"threads_started":1}
I20260924 20:22:53.208535 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:53.208128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:36961)
0924 20:22:53.208384 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:53.208443 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":69,"threads_started":1}
I20260924 20:22:53.450192 2041 raft_consensus.cc:482] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:22:53.450359 2041 raft_consensus.cc:504] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3a8a78c658a493ebc3d524249cfc30d" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34601 } } peers { permanent_uuid: "21f71bb592894c649988350dd5af12a8" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42773 } } peers { permanent_uuid: "1b2912cc83d344659f219f20e30df1f8" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 32847 } }
I20260924 20:22:53.450860 2041 leader_election.cc:288] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601), 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773)
W20260924 20:22:53.451624 1621 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21f71bb592894c649988350dd5af12a8 (127.18.172.67:42773): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
W20260924 20:22:53.451792 1622 leader_election.cc:341] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a3a8a78c658a493ebc3d524249cfc30d (127.18.172.66:34601): Not found: Tablet not found: 29b118fde6684e61b6f60c95f7319493
I20260924 20:22:53.451892 1622 leader_election.cc:302] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b2912cc83d344659f219f20e30df1f8; no voters: 21f71bb592894c649988350dd5af12a8, a3a8a78c658a493ebc3d524249cfc30d
I20260924 20:22:53.452054 2041 raft_consensus.cc:2726] T 29b118fde6684e61b6f60c95f7319493 P 1b2912cc83d344659f219f20e30df1f8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:22:53.866291 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:53.865885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:47009)
0924 20:22:53.866125 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:53.866192 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260924 20:22:54.018213 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:54.017635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:55731)
0924 20:22:54.018021 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:54.018086 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":84,"threads_started":1}
I20260924 20:22:54.209712 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:54.209318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:50043)
0924 20:22:54.209554 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:54.209614 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260924 20:22:54.867698 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:54.867229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:53417)
0924 20:22:54.867521 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:54.867591 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":71,"threads_started":1}
I20260924 20:22:55.019629 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:55.019154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:44263)
0924 20:22:55.019460 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:55.019523 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260924 20:22:55.211205 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:55.210594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:51069)
0924 20:22:55.210962 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:55.211062 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":87,"threads_started":1}
I20260924 20:22:55.868969 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:55.868512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:41777)
0924 20:22:55.868769 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:55.868871 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260924 20:22:56.021028 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:56.020549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:49387)
0924 20:22:56.020856 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:56.020924 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":80,"threads_started":1}
I20260924 20:22:56.212395 2050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:56.211961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:60063)
0924 20:22:56.212208 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:56.212300 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 20:22:56.870292 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:56.869771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:50483)
0924 20:22:56.870052 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:56.870117 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":77,"threads_started":1}
I20260924 20:22:57.022323 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:57.021885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:54595)
0924 20:22:57.022165 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:57.022229 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":75,"threads_started":1}
I20260924 20:22:57.213583 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:57.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.67:39283)
0924 20:22:57.213423 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:57.213489 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":133,"threads_started":1}
I20260924 20:22:57.871774 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:57.871370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.65:33505)
0924 20:22:57.871622 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:57.871678 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":81,"threads_started":1}
I20260924 20:22:58.023813 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:58.023337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:45629 (local address 127.18.172.66:41849)
0924 20:22:58.023629 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:58.023693 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:45629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":71,"threads_started":1}
I20260924 20:22:58.200376 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1569
I20260924 20:22:58.206192 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1737
I20260924 20:22:58.211763 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 1870
2026-09-24T20:22:58Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19760 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:22:58 dist-test-slave-m1qq krb5kdc[2058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:22:58 dist-test-slave-m1qq krb5kdc[2058](info): set up 2 sockets
Sep 24 20:22:58 dist-test-slave-m1qq krb5kdc[2058](info): commencing operation
krb5kdc: starting...
W20260924 20:22:59.276486 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:22:59 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:22:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:22:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:22:59.406422 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:59.517504 2071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:59.517848 2071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:59.517938 2071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:59.521997 2071 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:22:59.522089 2071 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:22:59.522116 2071 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:22:59.522141 2071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:22:59.522183 2071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:59.522225 2071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:59.522264 2071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:22:59.522312 2071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:22:59.527832 2071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:59.529021 2071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:59.530074 2071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:59.539785 2077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:59.539928 2071 server_base.cc:1034] running on GCE node
W20260924 20:22:59.540124 2079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:59.540136 2076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:59.540748 2071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:59.541850 2071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:59.543077 2071 hybrid_clock.cc:648] HybridClock initialized: now 1790281379543060 us; error 28 us; skew 500 ppm
Sep 24 20:22:59 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281379, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:59.546103 2071 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:22:59.546993 2071 webserver.cc:460] Webserver started at http://127.18.172.126:46335/ using document root <none> and password file <none>
I20260924 20:22:59.547569 2071 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:59.547627 2071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:59.547786 2071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:59.549794 2071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "87edb55bff344feb91085f2c982e848d"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "720781cda12098d0070890d62111b032"
server_key_iv: "5859bede3e3b58fb589b196a176ed3f3"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.550303 2071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "87edb55bff344feb91085f2c982e848d"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "720781cda12098d0070890d62111b032"
server_key_iv: "5859bede3e3b58fb589b196a176ed3f3"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.554003 2071 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:22:59.556437 2086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:59.557696 2071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:22:59.557794 2071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "87edb55bff344feb91085f2c982e848d"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "720781cda12098d0070890d62111b032"
server_key_iv: "5859bede3e3b58fb589b196a176ed3f3"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.557928 2071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:59.568964 2071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:59.572014 2071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:59.572168 2071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:59.580060 2071 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44809
I20260924 20:22:59.580415 2138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44809 every 8 connection(s)
I20260924 20:22:59.581089 2071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:22:59.582685 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2071
I20260924 20:22:59.582837 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:22:59.583199 19121 external_mini_cluster.cc:1428] Setting key 582dabe78b0ab2fa2d22bafc0b3b9a18
I20260924 20:22:59.584323 2139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:22:59.589900 2139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap starting.
Sep 24 20:22:59 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281379, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:22:59.592173 2139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Neither blocks nor log segments found. Creating new log.
I20260924 20:22:59.593026 2139 log.cc:824] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:22:59.595191 2139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: No bootstrap required, opened a new log
I20260924 20:22:59.597534 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:59.584609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.18.172.126:44809)
0924 20:22:59.585049 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:59.585066 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:59.585101 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:22:59.585742 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:59.585758 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:59.586084 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:59.586416 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:59.587272 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.588263 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.589146 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.589615 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.589794 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:59.592256 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:59.592261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:59.592293 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:59.592328 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:59.594364 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:59.594790 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:59.594796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:59.594801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:59.594953 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:59.595271 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:59.595274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:59.595275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:59.595792 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:59.595974 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:59.596222 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 20:22:59.596520 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":157,"threads_started":1}
I20260924 20:22:59.599630 2139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:22:59.599947 2139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:22:59.600039 2139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Initialized, Role: FOLLOWER
I20260924 20:22:59.600523 2139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:22:59.600675 2139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:22:59.600765 2139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:22:59.600864 2139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:22:59.601768 2139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:22:59.602103 2139 leader_election.cc:302] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87edb55bff344feb91085f2c982e848d; no voters:
I20260924 20:22:59.602434 2139 leader_election.cc:288] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:22:59.602566 2144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:22:59.602803 2144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 LEADER]: Becoming Leader. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Running, Role: LEADER
I20260924 20:22:59.603163 2144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:22:59.603734 2139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:22:59.604373 2144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87edb55bff344feb91085f2c982e848d. Latest consensus state: current_term: 1 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:22:59.604564 2144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:59.604887 2146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:22:59.605026 2146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:22:59.605064 2152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:22:59.607920 2152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:22:59.613425 2152 catalog_manager.cc:1347] Generated new cluster ID: e73d9cce14594e1ca123e01daaafcd3f
I20260924 20:22:59.613490 2152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:22:59.621008 2152 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:22:59.621887 2152 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:22:59.626518 2152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Generated new TSK 0
I20260924 20:22:59.627620 2152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:22:59.664112 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:59.778259 2165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:59.778515 2165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:59.778558 2165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:59.782527 2165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:59.782585 2165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:59.782641 2165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:22:59.787779 2165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:22:59.788772 2165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:22:59.789726 2165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:22:59.799476 2171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:22:59.799520 2173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:59.799666 2165 server_base.cc:1034] running on GCE node
W20260924 20:22:59.799476 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:22:59.800201 2165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:22:59.800858 2165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:22:59.802042 2165 hybrid_clock.cc:648] HybridClock initialized: now 1790281379802025 us; error 30 us; skew 500 ppm
Sep 24 20:22:59 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281379, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:22:59.805107 2165 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:22:59.806006 2165 webserver.cc:460] Webserver started at http://127.18.172.65:38739/ using document root <none> and password file <none>
I20260924 20:22:59.806596 2165 fs_manager.cc:359] Metadata directory not provided
I20260924 20:22:59.806682 2165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:22:59.806880 2165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:22:59.808858 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "fb368d712435426683140e8cf60b2260"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "f558d12817b1331075cd8a7715acebf5"
server_key_iv: "8c2714679da7a3cf59b6f064d635e17b"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.809453 2165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "fb368d712435426683140e8cf60b2260"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "f558d12817b1331075cd8a7715acebf5"
server_key_iv: "8c2714679da7a3cf59b6f064d635e17b"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.813325 2165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:22:59.815964 2180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:59.817319 2165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:22:59.817456 2165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fb368d712435426683140e8cf60b2260"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "f558d12817b1331075cd8a7715acebf5"
server_key_iv: "8c2714679da7a3cf59b6f064d635e17b"
server_key_version: "encryptionkey@0"
I20260924 20:22:59.817572 2165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:22:59.832237 2165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:22:59.835666 2165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:22:59.835889 2165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:22:59.836488 2165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:22:59.837316 2165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:22:59.837390 2165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:59.837464 2165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:22:59.837507 2165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:22:59.846165 2165 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46241
I20260924 20:22:59.846192 2293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46241 every 8 connection(s)
I20260924 20:22:59.847178 2165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:22:59.850927 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2165
I20260924 20:22:59.851080 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:22:59.851361 19121 external_mini_cluster.cc:1428] Setting key df72fb023d9b193a5fe7a05d3f86c1df
Sep 24 20:22:59 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281379, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:22:59.864408 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:59.849028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:43223 (local address 127.18.172.126:44809)
0924 20:22:59.849204 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 20:22:59.849209 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:22:59.850106 (+ 897us) server_negotiation.cc:407] Connection header received
0924 20:22:59.851985 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:22:59.851989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:22:59.852041 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:22:59.852140 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:22:59.853682 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.854545 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.856560 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.856793 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.856910 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:59.860005 (+ 3095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:22:59.860010 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:22:59.860016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:22:59.860049 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:22:59.862110 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:59.862752 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:59.862756 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:59.862759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:59.862861 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:22:59.863278 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:22:59.863281 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:22:59.863282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:22:59.863481 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:22:59.863566 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:22:59.864090 (+ 524us) server_negotiation.cc:299] Negotiation successful
0924 20:22:59.864222 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 20:22:59.865206 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:22:59.849410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:43223)
0924 20:22:59.849916 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 20:22:59.849967 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:22:59.851728 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:22:59.852336 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:22:59.852345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:22:59.853019 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:22:59.853524 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:59.853538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.854704 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.854711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:59.856434 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:22:59.856451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:22:59.856962 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:22:59.856969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:22:59.857243 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:22:59.857944 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:22:59.857966 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:22:59.859818 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:22:59.862309 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:59.862317 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:59.862332 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:59.862595 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:59.862967 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:22:59.862969 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:22:59.862971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:22:59.863154 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:22:59.863573 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:22:59.863578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:22:59.863806 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 20:22:59.864031 (+ 225us) client_negotiation.cc:240] Negotiation successful
0924 20:22:59.864359 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":106,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:22:59.866760 2294 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:22:59.867061 2294 heartbeater.cc:464] Registering TS with master...
I20260924 20:22:59.867547 2294 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
I20260924 20:22:59.869051 2103 ts_manager.cc:194] Registered new tserver with Master: fb368d712435426683140e8cf60b2260 (127.18.172.65:46241)
I20260924 20:22:59.870252 2103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:43223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:22:59.882165 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:22:59.993767 2299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:22:59.994089 2299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:22:59.994181 2299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:22:59.998453 2299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:22:59.998538 2299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:22:59.998628 2299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:23:00.004050 2299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:00.005300 2299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:00.006412 2299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:23:00.017040 2299 server_base.cc:1034] running on GCE node
W20260924 20:23:00.016983 2304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:00.016985 2307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:00.016980 2305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:00.017724 2299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:00.018478 2299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:00.019675 2299 hybrid_clock.cc:648] HybridClock initialized: now 1790281380019651 us; error 34 us; skew 500 ppm
Sep 24 20:23:00 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281380, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:00.028517 2299 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:23:00.029567 2299 webserver.cc:460] Webserver started at http://127.18.172.66:45437/ using document root <none> and password file <none>
I20260924 20:23:00.030143 2299 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:00.030206 2299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:00.030467 2299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:00.032804 2299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "a20d8a06d6924095837047b55f067e10"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "a92f98d29a8e81c030ded0c5fa107e2f"
server_key_iv: "2fba428f7e629181b77ba3037d561b97"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.033586 2299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "a20d8a06d6924095837047b55f067e10"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "a92f98d29a8e81c030ded0c5fa107e2f"
server_key_iv: "2fba428f7e629181b77ba3037d561b97"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.037716 2299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.005s
I20260924 20:23:00.040261 2314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.041596 2299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:23:00.041728 2299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a20d8a06d6924095837047b55f067e10"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "a92f98d29a8e81c030ded0c5fa107e2f"
server_key_iv: "2fba428f7e629181b77ba3037d561b97"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.041839 2299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:00.058391 2299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:00.061429 2299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:00.061635 2299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:00.062222 2299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:00.063047 2299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:00.063093 2299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.063133 2299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:00.063148 2299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.072417 2299 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:40971
I20260924 20:23:00.072434 2427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:40971 every 8 connection(s)
I20260924 20:23:00.073313 2299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:23:00.078785 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2299
I20260924 20:23:00.078933 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:23:00.079278 19121 external_mini_cluster.cc:1428] Setting key 8305b2f8b0a4abea1af4faefd03a5405
Sep 24 20:23:00 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281380, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:00.089658 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.075329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48143 (local address 127.18.172.126:44809)
0924 20:23:00.075489 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.075493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.076411 (+ 918us) server_negotiation.cc:407] Connection header received
0924 20:23:00.077844 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.077847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.077894 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.077974 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:00.079493 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.080272 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.081960 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.082209 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.082310 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.085326 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:00.085333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:00.085340 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:00.085377 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:00.087272 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.087912 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.087915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.087916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.087994 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.088450 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.088452 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.088454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.088661 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:00.088748 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.089429 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.089534 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:23:00.090801 2430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.075775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:48143)
0924 20:23:00.076258 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.076308 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.077549 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.078136 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.078143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.078540 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:00.079269 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.079291 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.080404 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.080407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.081801 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.081815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.082373 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.082380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.082623 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.083336 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:00.083352 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:00.085132 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:00.087402 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:00.087411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:00.087433 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:00.087807 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:00.088089 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:00.088094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:00.088097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:00.088297 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:00.088784 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:00.088793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:00.089152 (+ 359us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.089394 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.089784 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":119,"threads_started":1}
I20260924 20:23:00.092022 2428 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:23:00.092275 2428 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:00.092765 2428 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:23:00.094105 2103 ts_manager.cc:194] Registered new tserver with Master: a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:00.094686 2103 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:48143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:23:00.110253 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:00.220130 2433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:00.220381 2433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:00.220446 2433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:00.224785 2433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:00.224851 2433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:00.224941 2433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:23:00.230355 2433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:00.231568 2433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:00.232671 2433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:00.242973 2441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:00.243067 2438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:00.243181 2433 server_base.cc:1034] running on GCE node
W20260924 20:23:00.242973 2439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:00.243758 2433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:00.244387 2433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:00.245575 2433 hybrid_clock.cc:648] HybridClock initialized: now 1790281380245545 us; error 39 us; skew 500 ppm
Sep 24 20:23:00 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281380, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:00.248468 2433 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:23:00.249362 2433 webserver.cc:460] Webserver started at http://127.18.172.67:46819/ using document root <none> and password file <none>
I20260924 20:23:00.249938 2433 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:00.250000 2433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:00.250253 2433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:00.252426 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "8234a807d4004ee0b59ba77b9f3d2883"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "fb5c40bbc3a6643ad574d7ce031cec18"
server_key_iv: "57b900234208298e2885f255d4bbc25b"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.252988 2433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "8234a807d4004ee0b59ba77b9f3d2883"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "fb5c40bbc3a6643ad574d7ce031cec18"
server_key_iv: "57b900234208298e2885f255d4bbc25b"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.256742 2433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260924 20:23:00.259501 2448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.261160 2433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:23:00.261282 2433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "8234a807d4004ee0b59ba77b9f3d2883"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "fb5c40bbc3a6643ad574d7ce031cec18"
server_key_iv: "57b900234208298e2885f255d4bbc25b"
server_key_version: "encryptionkey@0"
I20260924 20:23:00.261399 2433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:00.270290 2433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:00.273267 2433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:00.273417 2433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:00.273931 2433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:00.274701 2433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:00.274745 2433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.274784 2433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:00.274801 2433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:00.283494 2433 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:34803
I20260924 20:23:00.283519 2561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:34803 every 8 connection(s)
I20260924 20:23:00.284433 2433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:23:00.286527 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2433
I20260924 20:23:00.286664 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:23:00.286962 19121 external_mini_cluster.cc:1428] Setting key d1766a91e98c4e10ff5efde42936c632
Sep 24 20:23:00 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281380, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:00.298758 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.286280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:46351 (local address 127.18.172.126:44809)
0924 20:23:00.286461 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.286467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.287445 (+ 978us) server_negotiation.cc:407] Connection header received
0924 20:23:00.288764 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.288767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.288823 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.288907 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:00.290215 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.290732 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.292074 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.292323 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.292418 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.294965 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:00.294968 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:00.294970 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:00.294994 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:00.296736 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.297274 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.297276 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.297278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.297365 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.297697 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.297699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.297701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.297921 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:00.298005 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.298532 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.298636 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 20:23:00.299643 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.286682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:46351)
0924 20:23:00.287256 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.287312 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.288584 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.289112 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.289121 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.289591 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:00.290073 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.290087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.290842 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.290845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.291799 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.291808 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.292470 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.292472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.292608 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.293211 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:00.293227 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:00.294786 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:00.296881 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:00.296887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:00.296899 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:00.297176 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:00.297455 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:00.297457 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:00.297458 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:00.297592 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:00.298017 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:00.298022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:00.298290 (+ 268us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.298479 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.298736 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":150,"threads_started":1}
I20260924 20:23:00.300863 2562 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:23:00.301105 2562 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:00.301599 2562 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
I20260924 20:23:00.302652 2101 ts_manager.cc:194] Registered new tserver with Master: 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803)
I20260924 20:23:00.303295 2101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:46351
I20260924 20:23:00.313712 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:23:00.323635 2142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.315829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.18.172.126:44809)
0924 20:23:00.315992 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.315997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.316311 (+ 314us) server_negotiation.cc:407] Connection header received
0924 20:23:00.316492 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.316497 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.316549 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.316614 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:00.317382 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.317877 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.318713 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.318940 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.319103 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.320412 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:00.320415 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:00.320418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:00.320440 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:00.322033 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.322486 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.322488 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.322489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.322561 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:00.322785 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:00.322787 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:00.322788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:00.322993 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:00.323155 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.323341 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.323441 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 20:23:00.326508 2101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:23:00.329530 2101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:23:00.343981 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.337900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34803 (local address 127.0.0.1:45040)
0924 20:23:00.338705 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.338756 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.338889 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.339914 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.339917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.339935 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:00.340023 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.340030 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.341072 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.341077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.342457 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.342466 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.343415 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.343526 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.343568 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.343686 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.343802 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":238,"thread_start_us":72,"threads_started":1}
I20260924 20:23:00.344019 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.338058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.18.172.67:34803)
0924 20:23:00.339490 (+ 1432us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.339501 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.339521 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:23:00.339591 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.339596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.339765 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.339931 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:00.340145 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.340936 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.342576 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.343284 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343396 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.343537 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.343729 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.343830 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1336,"thread_start_us":134,"threads_started":1}
I20260924 20:23:00.344561 2573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.338069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46241 (local address 127.0.0.1:38166)
0924 20:23:00.339120 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.339131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.339228 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.339707 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.339713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.339735 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:00.339890 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.339898 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.341442 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.341447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.342892 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.342904 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.344084 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.344087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.344203 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.344235 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.344320 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.344409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":159,"thread_start_us":116,"threads_started":1}
I20260924 20:23:00.344614 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.338290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.0.0.1:35228)
0924 20:23:00.339371 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.339382 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.339516 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.340090 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.340096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.340114 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:00.340204 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.340209 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.341534 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.341537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.342892 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.342909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.344159 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.344163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.344260 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.344311 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.344397 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.344464 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1032,"mutex_wait_us":55,"thread_start_us":91,"threads_started":1}
I20260924 20:23:00.344978 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.338673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.18.172.66:40971)
0924 20:23:00.339191 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.339204 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.339491 (+ 287us) server_negotiation.cc:407] Connection header received
0924 20:23:00.339675 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.339681 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.339877 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.340061 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:00.340315 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.341403 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343048 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.343856 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343958 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.344088 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.344705 (+ 617us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.344792 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":143,"threads_started":1}
I20260924 20:23:00.345331 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.338584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.18.172.65:46241)
0924 20:23:00.338972 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.338988 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.339222 (+ 234us) server_negotiation.cc:407] Connection header received
0924 20:23:00.339339 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.339346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.339560 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.339711 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:00.340037 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.341280 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343067 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.343796 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.343891 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.344017 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.344367 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.344463 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":70,"threads_started":1}
I20260924 20:23:00.347723 2362 tablet_service.cc:1467] Processing CreateTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a (DEFAULT_TABLE table=test-table [id=11ce78da776848bf824d86acf137a172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:00.347805 2228 tablet_service.cc:1467] Processing CreateTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a (DEFAULT_TABLE table=test-table [id=11ce78da776848bf824d86acf137a172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:00.347805 2496 tablet_service.cc:1467] Processing CreateTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a (DEFAULT_TABLE table=test-table [id=11ce78da776848bf824d86acf137a172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:00.348825 2228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aafad1adf6d4dec979d0a698fd86e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:00.348830 2496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aafad1adf6d4dec979d0a698fd86e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:00.348825 2362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aafad1adf6d4dec979d0a698fd86e5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:00.354895 2579 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Bootstrap starting.
I20260924 20:23:00.356134 2580 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Bootstrap starting.
I20260924 20:23:00.357909 2580 tablet_bootstrap.cc:654] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:00.358278 2579 tablet_bootstrap.cc:654] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:00.358995 2579 log.cc:824] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:00.358995 2580 log.cc:824] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:00.359383 2581 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Bootstrap starting.
I20260924 20:23:00.360996 2579 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: No bootstrap required, opened a new log
I20260924 20:23:00.361083 2580 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: No bootstrap required, opened a new log
I20260924 20:23:00.361256 2580 ts_tablet_manager.cc:1397] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:23:00.361220 2579 ts_tablet_manager.cc:1397] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:23:00.361608 2581 tablet_bootstrap.cc:654] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:00.362984 2581 log.cc:824] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:00.365259 2581 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: No bootstrap required, opened a new log
I20260924 20:23:00.365186 2579 raft_consensus.cc:348] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.365196 2580 raft_consensus.cc:348] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.365511 2579 raft_consensus.cc:374] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:00.365509 2580 raft_consensus.cc:374] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:00.365578 2580 raft_consensus.cc:729] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8234a807d4004ee0b59ba77b9f3d2883, State: Initialized, Role: FOLLOWER
I20260924 20:23:00.365607 2579 raft_consensus.cc:729] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20d8a06d6924095837047b55f067e10, State: Initialized, Role: FOLLOWER
I20260924 20:23:00.365520 2581 ts_tablet_manager.cc:1397] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260924 20:23:00.366086 2580 consensus_queue.cc:260] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.366261 2579 consensus_queue.cc:260] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.366842 2562 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
I20260924 20:23:00.367233 2580 ts_tablet_manager.cc:1428] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:23:00.367238 2579 ts_tablet_manager.cc:1428] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260924 20:23:00.367591 2428 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
I20260924 20:23:00.370214 2581 raft_consensus.cc:348] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.370543 2581 raft_consensus.cc:374] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:00.370651 2581 raft_consensus.cc:729] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb368d712435426683140e8cf60b2260, State: Initialized, Role: FOLLOWER
I20260924 20:23:00.371197 2581 consensus_queue.cc:260] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.371840 2294 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
I20260924 20:23:00.373013 2581 ts_tablet_manager.cc:1428] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:23:00.495239 2585 raft_consensus.cc:482] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:00.495448 2585 raft_consensus.cc:504] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.496718 2585 leader_election.cc:288] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb368d712435426683140e8cf60b2260 (127.18.172.65:46241), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:00.500988 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.497213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:38819 (local address 127.18.172.66:40971)
0924 20:23:00.497343 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.497350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.497575 (+ 225us) server_negotiation.cc:407] Connection header received
0924 20:23:00.497660 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.497663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.497709 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.497887 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:00.498193 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.498693 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.499942 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.500423 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.500552 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.500608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.500784 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.500865 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:23:00.500990 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.497069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.18.172.67:38819)
0924 20:23:00.497481 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.497493 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.497580 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.497940 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.497943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.497964 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:00.498074 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.498079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.498905 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.498909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.499747 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.499756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.500551 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.500555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.500655 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.500683 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.500742 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.500794 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":148,"threads_started":1}
I20260924 20:23:00.500998 2564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.496947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46241 (local address 127.18.172.67:47977)
0924 20:23:00.497104 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.497121 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:00.497217 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:00.497606 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:00.497610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:00.497637 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:00.497760 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.497768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.498692 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.498695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.499935 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:00.499945 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.500586 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.500590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:00.500705 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.500743 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:23:00.500831 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 20:23:00.500896 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 20:23:00.501062 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.497195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:47977 (local address 127.18.172.65:46241)
0924 20:23:00.497354 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.497358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.497372 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:23:00.497426 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.497429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.497477 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.497567 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:00.497975 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.498574 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.500065 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.500441 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.500576 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.500628 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.500860 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.500939 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 20:23:00.501691 2382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aafad1adf6d4dec979d0a698fd86e5a" candidate_uuid: "8234a807d4004ee0b59ba77b9f3d2883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20d8a06d6924095837047b55f067e10" is_pre_election: true
I20260924 20:23:00.501693 2248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aafad1adf6d4dec979d0a698fd86e5a" candidate_uuid: "8234a807d4004ee0b59ba77b9f3d2883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb368d712435426683140e8cf60b2260" is_pre_election: true
I20260924 20:23:00.502012 2248 raft_consensus.cc:2454] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8234a807d4004ee0b59ba77b9f3d2883 in term 0.
I20260924 20:23:00.502012 2382 raft_consensus.cc:2454] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8234a807d4004ee0b59ba77b9f3d2883 in term 0.
I20260924 20:23:00.502523 2452 leader_election.cc:302] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10; no voters:
I20260924 20:23:00.502786 2585 raft_consensus.cc:2781] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:23:00.502884 2585 raft_consensus.cc:482] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:23:00.502967 2585 raft_consensus.cc:3037] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:00.504165 2585 raft_consensus.cc:504] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.504586 2585 leader_election.cc:288] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [CANDIDATE]: Term 1 election: Requested vote from peers fb368d712435426683140e8cf60b2260 (127.18.172.65:46241), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:00.504946 2248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aafad1adf6d4dec979d0a698fd86e5a" candidate_uuid: "8234a807d4004ee0b59ba77b9f3d2883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb368d712435426683140e8cf60b2260"
I20260924 20:23:00.504966 2382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aafad1adf6d4dec979d0a698fd86e5a" candidate_uuid: "8234a807d4004ee0b59ba77b9f3d2883" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20d8a06d6924095837047b55f067e10"
I20260924 20:23:00.505093 2248 raft_consensus.cc:3037] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:00.505147 2382 raft_consensus.cc:3037] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:00.506271 2248 raft_consensus.cc:2454] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8234a807d4004ee0b59ba77b9f3d2883 in term 1.
I20260924 20:23:00.506428 2382 raft_consensus.cc:2454] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8234a807d4004ee0b59ba77b9f3d2883 in term 1.
I20260924 20:23:00.506615 2451 leader_election.cc:302] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8234a807d4004ee0b59ba77b9f3d2883, fb368d712435426683140e8cf60b2260; no voters:
I20260924 20:23:00.506903 2585 raft_consensus.cc:2781] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:23:00.507171 2585 raft_consensus.cc:686] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 LEADER]: Becoming Leader. State: Replica: 8234a807d4004ee0b59ba77b9f3d2883, State: Running, Role: LEADER
I20260924 20:23:00.507516 2585 consensus_queue.cc:237] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:00.510578 2103 catalog_manager.cc:5495] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "8234a807d4004ee0b59ba77b9f3d2883" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } health_report { overall_health: UNKNOWN } } }
W20260924 20:23:00.535328 2563 tablet.cc:2369] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:23:00.536120 2575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.532615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45056 (local address 127.18.172.67:34803)
0924 20:23:00.532752 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:00.532755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:00.532769 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:23:00.532904 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:00.532906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:00.532971 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:00.533064 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:23:00.533286 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.533777 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.534619 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:00.534821 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:00.534898 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:00.535229 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:23:00.535697 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:23:00.535837 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:00.535895 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 20:23:00.535958 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:23:00.541818 2248 raft_consensus.cc:1261] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Refusing update from remote peer 8234a807d4004ee0b59ba77b9f3d2883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:23:00.541818 2382 raft_consensus.cc:1261] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Refusing update from remote peer 8234a807d4004ee0b59ba77b9f3d2883: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:23:00.542485 2585 consensus_queue.cc:1035] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:23:00.542647 2589 consensus_queue.cc:1035] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:23:00.553170 2593 mvcc.cc:205] Tried to move back new op lower bound from 7332992534691831808 to 7332992534561374208. Current Snapshot: MvccSnapshot[applied={T|T < 7332992534691831808 or (T in {7332992534691831808})}]
I20260924 20:23:00.559151 2101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:23:00.559352 2101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:23:00.561673 2101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Sending DeleteTablet for 3 replicas of tablet 9aafad1adf6d4dec979d0a698fd86e5a
I20260924 20:23:00.562443 2496 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:45040
I20260924 20:23:00.562464 2228 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:38166
I20260924 20:23:00.562948 2362 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:35228
I20260924 20:23:00.563390 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2165
I20260924 20:23:00.563417 2603 tablet_replica.cc:330] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: stopping tablet replica
I20260924 20:23:00.563740 2603 raft_consensus.cc:2229] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:00.564011 2602 tablet_replica.cc:330] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: stopping tablet replica
I20260924 20:23:00.564265 2602 raft_consensus.cc:2229] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:23:00.564658 2602 raft_consensus.cc:2258] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:00.565644 2604 tablet_replica.cc:330] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: stopping tablet replica
I20260924 20:23:00.565954 2604 raft_consensus.cc:2229] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:00.566084 2602 ts_tablet_manager.cc:1905] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:23:00.566258 2604 raft_consensus.cc:2258] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:00.569064 2602 ts_tablet_manager.cc:1918] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:23:00.569154 2602 log.cc:1196] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/9aafad1adf6d4dec979d0a698fd86e5a
I20260924 20:23:00.569473 2602 ts_tablet_manager.cc:1939] T 9aafad1adf6d4dec979d0a698fd86e5a P 8234a807d4004ee0b59ba77b9f3d2883: Deleting consensus metadata
W20260924 20:23:00.570498 2089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:46241 (error 108)
I20260924 20:23:00.571066 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2299
I20260924 20:23:00.571144 2088 catalog_manager.cc:4841] TS 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): tablet 9aafad1adf6d4dec979d0a698fd86e5a (table test-table [id=11ce78da776848bf824d86acf137a172]) successfully deleted
I20260924 20:23:00.571955 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.571640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46241 (local address 127.0.0.1:38176)
0924 20:23:00.571786 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.571872 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 20:23:00.572692 2089 catalog_manager.cc:4584] TS fb368d712435426683140e8cf60b2260 (127.18.172.65:46241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aafad1adf6d4dec979d0a698fd86e5a: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46241: connect: Connection refused (error 111)
I20260924 20:23:00.577934 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2433
I20260924 20:23:00.578197 2574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:00.577865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.0.0.1:35236)
0924 20:23:00.578027 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:00.578096 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:40971: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 20:23:00.583554 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2071
I20260924 20:23:20.590584 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--webserver_interface=127.18.172.126
--webserver_port=46335
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:20.705389 2606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:20.705747 2606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:20.705826 2606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:20.710136 2606 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:23:20.710234 2606 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:23:20.710261 2606 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:23:20.710284 2606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:23:20.710326 2606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:20.710373 2606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:20.710430 2606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:23:20.710470 2606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:23:20.716233 2606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=46335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:20.717439 2606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:20.718528 2606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:20.732997 2612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:20.732997 2613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:20.733052 2606 server_base.cc:1034] running on GCE node
W20260924 20:23:20.732997 2615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:20.733927 2606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:20.735296 2606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:20.736487 2606 hybrid_clock.cc:648] HybridClock initialized: now 1790281400736446 us; error 55 us; skew 500 ppm
Sep 24 20:23:20 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281400, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:20.743880 2606 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:23:20.744925 2606 webserver.cc:460] Webserver started at http://127.18.172.126:46335/ using document root <none> and password file <none>
I20260924 20:23:20.745550 2606 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:20.745661 2606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:20.749943 2606 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:23:20.752377 2622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:20.753750 2606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:23:20.753904 2606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "87edb55bff344feb91085f2c982e848d"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "720781cda12098d0070890d62111b032"
server_key_iv: "5859bede3e3b58fb589b196a176ed3f3"
server_key_version: "encryptionkey@0"
I20260924 20:23:20.754575 2606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:20.773747 2606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:20.777064 2606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:20.777298 2606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:20.784961 2606 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44809
I20260924 20:23:20.785324 2674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44809 every 8 connection(s)
I20260924 20:23:20.786204 2606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:23:20.787885 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2606
I20260924 20:23:20.788375 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:46241
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=38739
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:23:20.791657 2675 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:23:20.793454 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap starting.
I20260924 20:23:20.810848 2675 log.cc:824] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:20.819977 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:23:20.820395 2675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap complete.
I20260924 20:23:20.824682 2675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:20.825119 2675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Initialized, Role: FOLLOWER
I20260924 20:23:20.825664 2675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:20.825785 2675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:23:20.825881 2675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:23:20.825999 2675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 1 FOLLOWER]: Advancing to term 2
I20260924 20:23:20.828123 2675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:20.828486 2675 leader_election.cc:302] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87edb55bff344feb91085f2c982e848d; no voters:
I20260924 20:23:20.828832 2675 leader_election.cc:288] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 20:23:20.828922 2679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Leader election won for term 2
I20260924 20:23:20.829157 2679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 LEADER]: Becoming Leader. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Running, Role: LEADER
I20260924 20:23:20.829528 2679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:20.830204 2675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:23:20.831115 2680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:23:20.831123 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87edb55bff344feb91085f2c982e848d. Latest consensus state: current_term: 2 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:23:20.831271 2680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:20.831278 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:20.831899 2688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:23:20.837059 2688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:23:20.837824 2688 catalog_manager.cc:1259] Loaded cluster ID: e73d9cce14594e1ca123e01daaafcd3f
I20260924 20:23:20.837894 2688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:23:20.840454 2688 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:23:20.841172 2688 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Loaded TSK: 0
I20260924 20:23:20.853286 2688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Generated new TSK 1
I20260924 20:23:20.853575 2688 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:23:20.910945 2677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:20.911278 2677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:20.911355 2677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:20.915637 2677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:20.915729 2677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:20.915825 2677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:23:20.921214 2677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:46241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=38739
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:20.922371 2677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:20.923470 2677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:20.934537 2703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:20.934538 2702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:20.934566 2705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:20.934860 2677 server_base.cc:1034] running on GCE node
I20260924 20:23:20.935393 2677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:20.936061 2677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:20.937258 2677 hybrid_clock.cc:648] HybridClock initialized: now 1790281400937235 us; error 39 us; skew 500 ppm
Sep 24 20:23:20 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281400, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:20.940523 2677 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:23:20.941628 2677 webserver.cc:460] Webserver started at http://127.18.172.65:38739/ using document root <none> and password file <none>
I20260924 20:23:20.942351 2677 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:20.942481 2677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:20.947088 2677 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:23:20.949595 2712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:20.951103 2677 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:23:20.951278 2677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fb368d712435426683140e8cf60b2260"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "f558d12817b1331075cd8a7715acebf5"
server_key_iv: "8c2714679da7a3cf59b6f064d635e17b"
server_key_version: "encryptionkey@0"
I20260924 20:23:20.951884 2677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:20.978230 2677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:20.981472 2677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:20.981694 2677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:20.982379 2677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:20.983675 2719 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:23:20.988644 2677 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:23:20.988750 2677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 20:23:20.988839 2677 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:23:20.990530 2677 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:23:20.990607 2677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:23:20.990885 2719 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Bootstrap starting.
I20260924 20:23:21.001168 2677 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46241
I20260924 20:23:21.001184 2826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46241 every 8 connection(s)
I20260924 20:23:21.002228 2677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:23:21.006073 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2677
I20260924 20:23:21.006598 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:40971
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=45437
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:23:21.010764 2719 log.cc:824] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281400, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:21.018666 2719 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:23:21.019153 2719 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Bootstrap complete.
I20260924 20:23:21.020449 2719 ts_tablet_manager.cc:1397] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.007s
I20260924 20:23:21.022646 2830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.004519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:53067)
0924 20:23:21.005120 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.005175 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.006033 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.007456 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.007465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.007905 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:21.008587 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.008612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.010467 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.010470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.011800 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.011816 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.012843 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.012849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.013095 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.013856 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:21.013882 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:21.015923 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:21.018875 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.018882 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.018900 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.019295 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.019747 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.019753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.019758 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.019936 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.020754 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:21.020760 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:21.021035 (+ 275us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.021369 (+ 334us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.021692 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":149,"threads_started":1}
I20260924 20:23:21.022750 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.004578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:53067 (local address 127.18.172.126:44809)
0924 20:23:21.004966 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.004986 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.005322 (+ 336us) server_negotiation.cc:407] Connection header received
0924 20:23:21.006549 (+ 1227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.006560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.006988 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.007396 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.008798 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.010283 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.011944 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.012684 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.012849 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.016163 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.016171 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.016194 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.016249 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.018713 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.019431 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.019440 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.019448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.019631 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.020058 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.020064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.020069 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.020623 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.020778 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.021532 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.021833 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":123,"threads_started":1}
I20260924 20:23:21.023468 2719 raft_consensus.cc:348] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.023891 2719 raft_consensus.cc:729] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb368d712435426683140e8cf60b2260, State: Initialized, Role: FOLLOWER
I20260924 20:23:21.024415 2719 consensus_queue.cc:260] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.025440 2719 ts_tablet_manager.cc:1428] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:23:21.025688 2827 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:23:21.025920 2827 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:21.026451 2827 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
I20260924 20:23:21.029397 2639 ts_manager.cc:194] Registered new tserver with Master: fb368d712435426683140e8cf60b2260 (127.18.172.65:46241)
I20260924 20:23:21.033015 2639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:53067
I20260924 20:23:21.035773 2827 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281400, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.65@KRBTEST.COM
I20260924 20:23:21.043434 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.032360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46241 (local address 127.0.0.1:43228)
0924 20:23:21.032889 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.032946 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.033078 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.034053 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.034060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.034499 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:21.034628 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.034635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.035961 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.035964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.036727 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.036736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.037322 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.037325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.037439 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.038065 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:21.038078 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:21.039678 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:21.041618 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.041629 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.041634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.041864 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.042178 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.042181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.042183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.042272 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.042739 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:21.042745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:21.042874 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.043034 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.043249 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":130,"threads_started":1}
I20260924 20:23:21.043434 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.033054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43228 (local address 127.18.172.65:46241)
0924 20:23:21.033442 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.033453 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.033475 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:23:21.033543 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.033547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.033760 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.033907 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.034764 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.035818 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.036847 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.037198 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.037317 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.039824 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.039839 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.039849 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.039870 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.041499 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.041996 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.042001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.042002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.042077 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.042387 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.042390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.042391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.042618 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.042765 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.043117 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.043277 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":111,"threads_started":1}
I20260924 20:23:21.044467 2761 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260924 20:23:21.045509 2836 tablet_replica.cc:330] stopping tablet replica
I20260924 20:23:21.046006 2836 raft_consensus.cc:2229] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:21.046398 2836 raft_consensus.cc:2258] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:21.047588 2836 ts_tablet_manager.cc:1905] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:23:21.051534 2836 ts_tablet_manager.cc:1918] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:23:21.051666 2836 log.cc:1196] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal/wals/9aafad1adf6d4dec979d0a698fd86e5a
I20260924 20:23:21.051973 2836 ts_tablet_manager.cc:1939] T 9aafad1adf6d4dec979d0a698fd86e5a P fb368d712435426683140e8cf60b2260: Deleting consensus metadata
I20260924 20:23:21.053886 2625 catalog_manager.cc:4841] TS fb368d712435426683140e8cf60b2260 (127.18.172.65:46241): tablet 9aafad1adf6d4dec979d0a698fd86e5a (table test-table [id=11ce78da776848bf824d86acf137a172]) successfully deleted
W20260924 20:23:21.130267 2831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:21.130599 2831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:21.130671 2831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:21.135110 2831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:21.135164 2831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:21.135264 2831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:23:21.140520 2831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:40971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=45437
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:21.141567 2831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:21.142612 2831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:21.152527 2844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:21.152720 2841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:21.152571 2842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:21.152557 2831 server_base.cc:1034] running on GCE node
I20260924 20:23:21.153218 2831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:21.153874 2831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:21.155073 2831 hybrid_clock.cc:648] HybridClock initialized: now 1790281401155045 us; error 46 us; skew 500 ppm
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:21.157989 2831 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:23:21.158985 2831 webserver.cc:460] Webserver started at http://127.18.172.66:45437/ using document root <none> and password file <none>
I20260924 20:23:21.159729 2831 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:21.159813 2831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:21.163702 2831 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260924 20:23:21.165930 2851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:21.167496 2831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:23:21.167639 2831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a20d8a06d6924095837047b55f067e10"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "a92f98d29a8e81c030ded0c5fa107e2f"
server_key_iv: "2fba428f7e629181b77ba3037d561b97"
server_key_version: "encryptionkey@0"
I20260924 20:23:21.168277 2831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:21.196645 2831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:21.200101 2831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:21.200325 2831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:21.200949 2831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:21.202109 2858 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:23:21.206507 2831 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:23:21.206596 2831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260924 20:23:21.206676 2831 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:23:21.208395 2831 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:23:21.208468 2831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260924 20:23:21.208771 2858 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Bootstrap starting.
I20260924 20:23:21.214432 2858 log.cc:824] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:21.218807 2831 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:40971
I20260924 20:23:21.218870 2966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:40971 every 8 connection(s)
I20260924 20:23:21.219659 2831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:23:21.222452 2858 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:23:21.222847 2858 tablet_bootstrap.cc:492] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Bootstrap complete.
I20260924 20:23:21.223238 2858 ts_tablet_manager.cc:1397] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20260924 20:23:21.226192 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2831
I20260924 20:23:21.226030 2858 raft_consensus.cc:348] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.226583 2858 raft_consensus.cc:729] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20d8a06d6924095837047b55f067e10, State: Initialized, Role: FOLLOWER
I20260924 20:23:21.226686 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:34803
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=46819
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:23:21.227109 2858 consensus_queue.cc:260] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.228185 2858 ts_tablet_manager.cc:1428] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:21.235508 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.221522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:41543 (local address 127.18.172.126:44809)
0924 20:23:21.221689 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.221693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.222335 (+ 642us) server_negotiation.cc:407] Connection header received
0924 20:23:21.223448 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.223454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.223514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.223625 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.224707 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.225536 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.226907 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.227221 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.227323 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.231163 (+ 3840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.231166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.231169 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.231195 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.232990 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.233689 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.233692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.233693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.233791 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.234257 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.234259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.234260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.234436 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.234511 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.235268 (+ 757us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.235383 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:23:21.236969 2969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.221658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:41543)
0924 20:23:21.222155 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.222213 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.223212 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.223749 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.223755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.224086 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:21.224542 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.224555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.225680 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.225683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.226736 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.226764 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.227516 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.227521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.227726 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.228829 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:21.228853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:21.230934 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:21.233125 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.233135 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.233155 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.233581 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.233893 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.233899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.233904 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.234137 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.234546 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:21.234557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:21.234933 (+ 376us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.235237 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.235680 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260924 20:23:21.238363 2967 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:23:21.238633 2967 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:21.239213 2967 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
I20260924 20:23:21.241312 2639 ts_manager.cc:194] Registered new tserver with Master: a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:21.242786 2639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:41543
I20260924 20:23:21.245292 2967 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281400, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.66@KRBTEST.COM
I20260924 20:23:21.254246 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.242470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.0.0.1:42960)
0924 20:23:21.242665 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.242681 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.242764 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.243584 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.243588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.243826 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:21.243938 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.243946 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.245396 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.245403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.246465 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.246476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.247460 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.247463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.247550 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.248148 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:21.248160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:21.249732 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:21.252425 (+ 2693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.252428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.252429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.252650 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.253021 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.253024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.253025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.253112 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.253671 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:21.253675 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:21.253779 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.253891 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.254053 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 20:23:21.254336 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.242557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42960 (local address 127.18.172.66:40971)
0924 20:23:21.243052 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.243065 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.243089 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:23:21.243173 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.243178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.243375 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.243572 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.244079 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.245251 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.246644 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.247322 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.247435 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.249890 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.249904 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.249922 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.249954 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.252290 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.252772 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.252779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.252781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.252883 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.253278 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.253283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.253285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.253544 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.253687 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.253997 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.254176 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":351,"thread_start_us":142,"threads_started":1}
I20260924 20:23:21.255431 2899 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aafad1adf6d4dec979d0a698fd86e5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:42960
I20260924 20:23:21.256592 2973 tablet_replica.cc:330] stopping tablet replica
I20260924 20:23:21.256887 2973 raft_consensus.cc:2229] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:21.257153 2973 raft_consensus.cc:2258] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:21.258335 2973 ts_tablet_manager.cc:1905] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:23:21.262367 2973 ts_tablet_manager.cc:1918] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:23:21.262483 2973 log.cc:1196] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/9aafad1adf6d4dec979d0a698fd86e5a
I20260924 20:23:21.262759 2973 ts_tablet_manager.cc:1939] T 9aafad1adf6d4dec979d0a698fd86e5a P a20d8a06d6924095837047b55f067e10: Deleting consensus metadata
I20260924 20:23:21.264572 2626 catalog_manager.cc:4841] TS a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): tablet 9aafad1adf6d4dec979d0a698fd86e5a (table test-table [id=11ce78da776848bf824d86acf137a172]) successfully deleted
W20260924 20:23:21.352486 2971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:21.352737 2971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:21.352785 2971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:21.356825 2971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:21.356885 2971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:21.356946 2971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:23:21.362653 2971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=46819
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:21.363796 2971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:21.365027 2971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:21.375759 2978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:21.375759 2979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:21.375806 2981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:21.376016 2971 server_base.cc:1034] running on GCE node
I20260924 20:23:21.376462 2971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:21.377136 2971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:21.378295 2971 hybrid_clock.cc:648] HybridClock initialized: now 1790281401378275 us; error 32 us; skew 500 ppm
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:21.381131 2971 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:23:21.381965 2971 webserver.cc:460] Webserver started at http://127.18.172.67:46819/ using document root <none> and password file <none>
I20260924 20:23:21.382500 2971 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:21.382568 2971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:21.386492 2971 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:23:21.388729 2988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:21.389976 2971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:23:21.390081 2971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "8234a807d4004ee0b59ba77b9f3d2883"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "fb5c40bbc3a6643ad574d7ce031cec18"
server_key_iv: "57b900234208298e2885f255d4bbc25b"
server_key_version: "encryptionkey@0"
I20260924 20:23:21.390656 2971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:21.423489 2971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:21.426589 2971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:21.426810 2971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:21.427455 2971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:21.428308 2971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:21.428355 2971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:21.428427 2971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:21.428471 2971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:21.438164 2971 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:34803
I20260924 20:23:21.438194 3101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:34803 every 8 connection(s)
I20260924 20:23:21.439205 2971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:23:21.445091 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 2971
Sep 24 20:23:21 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281401, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:21.454175 2829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.441105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:49101 (local address 127.18.172.126:44809)
0924 20:23:21.441267 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.441272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.442153 (+ 881us) server_negotiation.cc:407] Connection header received
0924 20:23:21.443479 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.443482 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.443530 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.443604 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.444735 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.445526 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.446886 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.447305 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.447432 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.450358 (+ 2926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.450361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.450363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.450386 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.451933 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.452554 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.452557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.452558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.452634 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.453015 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.453017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.453018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.453200 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.453304 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.453894 (+ 590us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.454026 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":2944}
I20260924 20:23:21.455004 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.445811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.18.172.126:44809)
0924 20:23:21.446073 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.446078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.446286 (+ 208us) server_negotiation.cc:407] Connection header received
0924 20:23:21.446458 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.446461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.446502 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.446593 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.447501 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.448243 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.449246 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.449469 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.449545 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.451444 (+ 1899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.451448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.451450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.451488 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.453212 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.453636 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.453640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.453642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.453734 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.454027 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.454030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.454032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.454243 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.454426 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.454753 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.454849 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"spinlock_wait_cycles":768,"thread_start_us":107,"threads_started":1}
I20260924 20:23:21.455255 3104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.441549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:49101)
0924 20:23:21.441995 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.442038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.443275 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.443740 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.443747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.444099 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:21.444589 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.444603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.445678 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.445682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.446766 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.446776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.447487 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.447492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.447702 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.448425 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:21.448442 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:21.450133 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:21.452028 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.452036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.452056 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.452451 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.452732 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:21.452736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:21.452738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:21.452915 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:21.453313 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:21.453322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:21.453612 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.453829 (+ 217us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.454196 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":106,"threads_started":1}
I20260924 20:23:21.456321 3102 heartbeater.cc:347] Connected to a master server at 127.18.172.126:44809
I20260924 20:23:21.456563 3102 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:21.457123 3102 heartbeater.cc:511] Master 127.18.172.126:44809 requested a full tablet report, sending...
I20260924 20:23:21.458148 2639 ts_manager.cc:194] Registered new tserver with Master: 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803)
I20260924 20:23:21.458757 2639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:49101
I20260924 20:23:21.469043 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:23:21.473791 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.469773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.18.172.126:44809)
0924 20:23:21.469902 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.469907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.470069 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:23:21.470292 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.470295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.470401 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.470491 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:23:21.470771 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.471435 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.472524 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.472802 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.472907 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.473303 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:23:21.473458 (+ 155us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 20:23:21.473682 (+ 224us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33198: token expired
Metrics: {"server-negotiator.queue_time_us":46}
W20260924 20:23:21.473896 3105 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33198: token expired
W20260924 20:23:21.474296 2568 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.18.172.126:44809)
I20260924 20:23:21.481851 3105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.474721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33206 (local address 127.18.172.126:44809)
0924 20:23:21.474838 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.474841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.474852 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:23:21.474990 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.474993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.475055 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.475117 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:21.475813 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.476311 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.477158 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.477365 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.477440 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.478911 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:21.478914 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:21.478916 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:21.478939 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:21.480379 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.480780 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.480783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.480784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.480855 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:21.481084 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:21.481085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:21.481087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:21.481306 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:21.481404 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.481635 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.481729 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:23:21.484670 2639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:23:21.488130 2639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:23:21.496170 2761 tablet_service.cc:1467] Processing CreateTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 (DEFAULT_TABLE table=test-table [id=ada6857b66d9445fb1be401d04582546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:21.496170 2899 tablet_service.cc:1467] Processing CreateTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 (DEFAULT_TABLE table=test-table [id=ada6857b66d9445fb1be401d04582546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:21.496853 2899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abc6e93516f47668f6c63cd2d5a8734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:21.496853 2761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abc6e93516f47668f6c63cd2d5a8734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:21.499938 2858 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Bootstrap starting.
I20260924 20:23:21.500265 2719 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Bootstrap starting.
I20260924 20:23:21.500377 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.495392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34803 (local address 127.0.0.1:52058)
0924 20:23:21.495735 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.495753 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.495839 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.496575 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.496577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.496599 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:21.496706 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.496711 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.498183 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.498187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.498981 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.498990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.499939 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.499943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.500050 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.500108 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.500167 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.500247 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220}
I20260924 20:23:21.501025 2858 tablet_bootstrap.cc:654] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:21.501044 2719 tablet_bootstrap.cc:654] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:21.501098 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.495660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52058 (local address 127.18.172.67:34803)
0924 20:23:21.496089 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.496100 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.496120 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:23:21.496185 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.496189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.496397 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.496843 (+ 446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:21.497049 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.498057 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.499168 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.499789 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.500582 (+ 793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.500755 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.500812 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.500924 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":174,"threads_started":1}
I20260924 20:23:21.502609 2858 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: No bootstrap required, opened a new log
I20260924 20:23:21.502728 2858 ts_tablet_manager.cc:1397] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 20:23:21.503295 2719 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: No bootstrap required, opened a new log
I20260924 20:23:21.503384 2719 ts_tablet_manager.cc:1397] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260924 20:23:21.503315 2858 raft_consensus.cc:348] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.503433 2858 raft_consensus.cc:374] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:21.503473 2858 raft_consensus.cc:729] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a20d8a06d6924095837047b55f067e10, State: Initialized, Role: FOLLOWER
I20260924 20:23:21.503616 2858 consensus_queue.cc:260] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.503891 2858 ts_tablet_manager.cc:1428] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 20:23:21.503844 2719 raft_consensus.cc:348] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.503952 2719 raft_consensus.cc:374] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:21.503991 2719 raft_consensus.cc:729] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb368d712435426683140e8cf60b2260, State: Initialized, Role: FOLLOWER
I20260924 20:23:21.504127 2719 consensus_queue.cc:260] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.504983 2719 ts_tablet_manager.cc:1428] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 20:23:21.504596 3036 tablet_service.cc:1467] Processing CreateTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 (DEFAULT_TABLE table=test-table [id=ada6857b66d9445fb1be401d04582546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:21.505885 3036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abc6e93516f47668f6c63cd2d5a8734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:21.513375 3114 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Bootstrap starting.
I20260924 20:23:21.515754 3114 tablet_bootstrap.cc:654] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:21.516865 3114 log.cc:824] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:21.518682 3114 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: No bootstrap required, opened a new log
I20260924 20:23:21.518903 3114 ts_tablet_manager.cc:1397] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:23:21.522171 3114 raft_consensus.cc:348] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.522408 3114 raft_consensus.cc:374] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:21.522478 3114 raft_consensus.cc:729] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8234a807d4004ee0b59ba77b9f3d2883, State: Initialized, Role: FOLLOWER
I20260924 20:23:21.522945 3114 consensus_queue.cc:260] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.523720 3102 heartbeater.cc:503] Master 127.18.172.126:44809 was elected leader, sending a full tablet report...
I20260924 20:23:21.524005 3114 ts_tablet_manager.cc:1428] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260924 20:23:21.690234 3103 tablet.cc:2369] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:23:21.695523 2833 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:21.695736 2833 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.696846 2833 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:21.700716 3111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.697186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:53419 (local address 127.18.172.67:34803)
0924 20:23:21.697367 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.697374 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.697484 (+ 110us) server_negotiation.cc:407] Connection header received
0924 20:23:21.697698 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.697701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.697744 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.697832 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:21.698092 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.698588 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.699741 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.700140 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.700284 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.700338 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.700524 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.700594 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 20:23:21.700745 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.697052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34803 (local address 127.18.172.65:53419)
0924 20:23:21.697368 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.697384 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.697552 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.697846 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.697849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.697866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:21.697978 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.697983 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.698710 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.698716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.699612 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.699621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.700277 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.700280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.700366 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.700417 (+ 51us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.700507 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.700581 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":63,"thread_start_us":81,"threads_started":1}
I20260924 20:23:21.700848 2972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.697259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:36809 (local address 127.18.172.66:40971)
0924 20:23:21.697412 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.697417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.697623 (+ 206us) server_negotiation.cc:407] Connection header received
0924 20:23:21.697790 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.697792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.697840 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.697941 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:21.698196 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.698746 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.699882 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.700338 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.700468 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.700609 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.700643 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.700717 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:23:21.700903 3118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.697156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.18.172.65:36809)
0924 20:23:21.697506 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.697518 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:21.697658 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:21.697955 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:21.697958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:21.697974 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:21.698067 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.698074 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.698871 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.698876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.699753 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:21.699763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.700462 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.700466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:21.700545 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.700565 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 20:23:21.700628 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 20:23:21.700698 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":43,"thread_start_us":62,"threads_started":1}
I20260924 20:23:21.701445 3056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abc6e93516f47668f6c63cd2d5a8734" candidate_uuid: "fb368d712435426683140e8cf60b2260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8234a807d4004ee0b59ba77b9f3d2883" is_pre_election: true
I20260924 20:23:21.701442 2920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abc6e93516f47668f6c63cd2d5a8734" candidate_uuid: "fb368d712435426683140e8cf60b2260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20d8a06d6924095837047b55f067e10" is_pre_election: true
I20260924 20:23:21.701745 2920 raft_consensus.cc:2454] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb368d712435426683140e8cf60b2260 in term 0.
I20260924 20:23:21.701754 3056 raft_consensus.cc:2454] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb368d712435426683140e8cf60b2260 in term 0.
I20260924 20:23:21.702229 2716 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a20d8a06d6924095837047b55f067e10, fb368d712435426683140e8cf60b2260; no voters:
I20260924 20:23:21.702520 2833 raft_consensus.cc:2781] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:23:21.702606 2833 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:23:21.702648 2833 raft_consensus.cc:3037] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:21.703780 2833 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.704120 2833 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 1 election: Requested vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
I20260924 20:23:21.704522 3056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abc6e93516f47668f6c63cd2d5a8734" candidate_uuid: "fb368d712435426683140e8cf60b2260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8234a807d4004ee0b59ba77b9f3d2883"
I20260924 20:23:21.704522 2920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abc6e93516f47668f6c63cd2d5a8734" candidate_uuid: "fb368d712435426683140e8cf60b2260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a20d8a06d6924095837047b55f067e10"
I20260924 20:23:21.704674 3056 raft_consensus.cc:3037] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:21.704716 2920 raft_consensus.cc:3037] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:21.706110 2920 raft_consensus.cc:2454] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb368d712435426683140e8cf60b2260 in term 1.
I20260924 20:23:21.706247 3056 raft_consensus.cc:2454] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb368d712435426683140e8cf60b2260 in term 1.
I20260924 20:23:21.706476 2716 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a20d8a06d6924095837047b55f067e10, fb368d712435426683140e8cf60b2260; no voters:
I20260924 20:23:21.706696 2833 raft_consensus.cc:2781] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:23:21.706879 2833 raft_consensus.cc:686] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 LEADER]: Becoming Leader. State: Replica: fb368d712435426683140e8cf60b2260, State: Running, Role: LEADER
I20260924 20:23:21.707193 2833 consensus_queue.cc:237] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:21.710397 2637 catalog_manager.cc:5495] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb368d712435426683140e8cf60b2260 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "fb368d712435426683140e8cf60b2260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } health_report { overall_health: UNKNOWN } } }
W20260924 20:23:21.720865 2968 tablet.cc:2369] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:23:21.741726 3122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.738009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.18.172.65:46241)
0924 20:23:21.738309 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:21.738315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:21.738327 (+ 12us) server_negotiation.cc:407] Connection header received
0924 20:23:21.738402 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:21.738404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:21.738455 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:21.738588 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:23:21.738841 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.739420 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.740274 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:21.740494 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:21.740600 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:21.740887 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:23:21.741331 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:23:21.741464 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:21.741504 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 20:23:21.741557 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":99,"threads_started":1}
I20260924 20:23:21.745770 2920 raft_consensus.cc:1261] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Refusing update from remote peer fb368d712435426683140e8cf60b2260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:23:21.745775 3056 raft_consensus.cc:1261] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Refusing update from remote peer fb368d712435426683140e8cf60b2260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:23:21.746483 3119 consensus_queue.cc:1035] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:23:21.746739 2833 consensus_queue.cc:1035] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:23:21.752831 3124 mvcc.cc:205] Tried to move back new op lower bound from 7332992621544349696 to 7332992621395537920. Current Snapshot: MvccSnapshot[applied={T|T < 7332992621544349696}]
I20260924 20:23:21.754083 3125 mvcc.cc:205] Tried to move back new op lower bound from 7332992621544349696 to 7332992621395537920. Current Snapshot: MvccSnapshot[applied={T|T < 7332992621544349696}]
I20260924 20:23:21.762179 2637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:23:21.762362 2637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33206:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:23:21.764994 2637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Sending DeleteTablet for 3 replicas of tablet 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:21.765846 2899 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:21 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:42960
I20260924 20:23:21.765869 3036 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:21 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:52058
I20260924 20:23:21.766171 2761 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abc6e93516f47668f6c63cd2d5a8734 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:23:21 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:43228
I20260924 20:23:21.766355 3133 tablet_replica.cc:330] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: stopping tablet replica
I20260924 20:23:21.766479 3133 raft_consensus.cc:2229] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:21.766563 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2677
I20260924 20:23:21.768324 3134 tablet_replica.cc:330] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: stopping tablet replica
I20260924 20:23:21.768694 3134 raft_consensus.cc:2229] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:23:21.769012 3134 raft_consensus.cc:2258] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:21.769390 3133 raft_consensus.cc:2258] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:23:21.770320 3133 ts_tablet_manager.cc:1905] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:23:21.770560 3134 ts_tablet_manager.cc:1905] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:23:21.773077 3133 ts_tablet_manager.cc:1918] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260924 20:23:21.773082 2625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:46241 (error 108)
I20260924 20:23:21.773180 3133 log.cc:1196] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:21.773295 3134 ts_tablet_manager.cc:1918] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:23:21.773394 3134 log.cc:1196] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:21.773524 3133 ts_tablet_manager.cc:1939] T 5abc6e93516f47668f6c63cd2d5a8734 P a20d8a06d6924095837047b55f067e10: Deleting consensus metadata
I20260924 20:23:21.773890 3134 ts_tablet_manager.cc:1939] T 5abc6e93516f47668f6c63cd2d5a8734 P 8234a807d4004ee0b59ba77b9f3d2883: Deleting consensus metadata
I20260924 20:23:21.774397 2834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:21.774057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:46241 (local address 127.0.0.1:43238)
0924 20:23:21.774222 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:21.774306 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46241: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:23:21.774398 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2831
I20260924 20:23:21.774444 2626 catalog_manager.cc:4841] TS a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): tablet 5abc6e93516f47668f6c63cd2d5a8734 (table test-table [id=ada6857b66d9445fb1be401d04582546]) successfully deleted
W20260924 20:23:21.774677 2625 catalog_manager.cc:4584] TS fb368d712435426683140e8cf60b2260 (127.18.172.65:46241): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5abc6e93516f47668f6c63cd2d5a8734: Network error: Client connection negotiation failed: client connection to 127.18.172.65:46241: connect: Connection refused (error 111)
I20260924 20:23:21.775895 2624 catalog_manager.cc:4841] TS 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): tablet 5abc6e93516f47668f6c63cd2d5a8734 (table test-table [id=ada6857b66d9445fb1be401d04582546]) successfully deleted
I20260924 20:23:21.780663 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2971
I20260924 20:23:21.786679 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 2606
I20260924 20:23:41.793511 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--webserver_interface=127.18.172.126
--webserver_port=46335
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:41.908097 3137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:41.908385 3137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:41.908427 3137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:41.912554 3137 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 20:23:41.912612 3137 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 20:23:41.912629 3137 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 20:23:41.912643 3137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:23:41.912662 3137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:41.912680 3137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:41.912698 3137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:23:41.912714 3137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:23:41.918145 3137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44809
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=46335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:41.919250 3137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:41.920284 3137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:41.929167 3145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:41.929100 3142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:41.929144 3137 server_base.cc:1034] running on GCE node
W20260924 20:23:41.929100 3143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:41.930105 3137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:41.931277 3137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:41.932482 3137 hybrid_clock.cc:648] HybridClock initialized: now 1790281421932461 us; error 33 us; skew 500 ppm
Sep 24 20:23:41 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281421, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:41.939327 3137 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:23:41.940323 3137 webserver.cc:460] Webserver started at http://127.18.172.126:46335/ using document root <none> and password file <none>
I20260924 20:23:41.941076 3137 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:41.941188 3137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:41.945560 3137 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:23:41.947902 3152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:41.949126 3137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:23:41.949276 3137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "87edb55bff344feb91085f2c982e848d"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "720781cda12098d0070890d62111b032"
server_key_iv: "5859bede3e3b58fb589b196a176ed3f3"
server_key_version: "encryptionkey@0"
I20260924 20:23:41.949928 3137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:41.963359 3137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:41.966472 3137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:41.966694 3137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:41.974372 3137 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44809
I20260924 20:23:41.974681 3204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44809 every 8 connection(s)
I20260924 20:23:41.975519 3137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:23:41.979877 3205 sys_catalog.cc:263] Verifying existing consensus state
I20260924 20:23:41.980374 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3137
I20260924 20:23:41.980826 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:46241
--local_ip_for_outbound_sockets=127.18.172.65
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=38739
--webserver_interface=127.18.172.65
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:23:41.981634 3205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap starting.
I20260924 20:23:41.999869 3205 log.cc:824] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:42.011200 3205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:23:42.011616 3205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Bootstrap complete.
I20260924 20:23:42.016018 3205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:42.016396 3205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Initialized, Role: FOLLOWER
I20260924 20:23:42.016883 3205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:42.017002 3205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:23:42.017091 3205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:23:42.017206 3205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 2 FOLLOWER]: Advancing to term 3
I20260924 20:23:42.019150 3205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:42.019472 3205 leader_election.cc:302] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87edb55bff344feb91085f2c982e848d; no voters:
I20260924 20:23:42.019799 3205 leader_election.cc:288] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 20:23:42.019892 3209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 3 FOLLOWER]: Leader election won for term 3
I20260924 20:23:42.020092 3209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [term 3 LEADER]: Becoming Leader. State: Replica: 87edb55bff344feb91085f2c982e848d, State: Running, Role: LEADER
I20260924 20:23:42.020452 3209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } }
I20260924 20:23:42.021036 3205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:23:42.021343 3209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87edb55bff344feb91085f2c982e848d. Latest consensus state: current_term: 3 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:23:42.021550 3209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:42.021714 3210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "87edb55bff344feb91085f2c982e848d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87edb55bff344feb91085f2c982e848d" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44809 } } }
I20260924 20:23:42.021890 3210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:42.021966 3213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:23:42.027076 3213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:23:42.027786 3213 catalog_manager.cc:1259] Loaded cluster ID: e73d9cce14594e1ca123e01daaafcd3f
I20260924 20:23:42.027863 3213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:23:42.029646 3213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:23:42.030259 3213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Loaded TSK: 0
I20260924 20:23:42.030349 3213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 87edb55bff344feb91085f2c982e848d: Loaded TSK: 1
F20260924 20:23:42.030948 3213 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f55380049ad google::LogMessage::Fail() at ??:0
@ 0x7f553800788a google::LogMessage::SendToLog() at ??:0
@ 0x7f55380044de google::LogMessage::Flush() at ??:0
@ 0x7f5538005a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f553acc818d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f553acc0f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f554296d7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f554294799c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5542948178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f5542989df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f55429a882b std::function<>::operator()() at ??:0
@ 0x7f5542947b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5542948f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5542945579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5542989640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f553a2563d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f553a256c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f553a2583f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f553a247512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f553b88b6db start_thread at ??:0
@ 0x7f5536b0261f clone at ??:0
W20260924 20:23:42.116060 3207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:42.116294 3207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:42.116338 3207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:42.120864 3207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:42.120954 3207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:42.121019 3207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:23:42.127137 3207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:46241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=38739
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:42.128162 3207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:42.129134 3207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:42.139235 3253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:42.139248 3255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:42.139253 3252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:42.140074 3207 server_base.cc:1034] running on GCE node
I20260924 20:23:42.140515 3207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:42.141176 3207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:42.142351 3207 hybrid_clock.cc:648] HybridClock initialized: now 1790281422142307 us; error 58 us; skew 500 ppm
Sep 24 20:23:42 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281422, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:42.145576 3207 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:23:42.146513 3207 webserver.cc:460] Webserver started at http://127.18.172.65:38739/ using document root <none> and password file <none>
I20260924 20:23:42.147164 3207 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:42.147230 3207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:42.151336 3207 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:23:42.153491 3262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.154851 3207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:23:42.154946 3207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "fb368d712435426683140e8cf60b2260"
format_stamp: "Formatted at 2026-09-24 20:22:59 on dist-test-slave-m1qq"
server_key: "f558d12817b1331075cd8a7715acebf5"
server_key_iv: "8c2714679da7a3cf59b6f064d635e17b"
server_key_version: "encryptionkey@0"
I20260924 20:23:42.155529 3207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:42.169327 3207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:42.172823 3207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:42.173056 3207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:42.173744 3207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:42.174866 3269 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 20:23:42.180636 3207 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 20:23:42.180724 3207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260924 20:23:42.180805 3207 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 20:23:42.182521 3207 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 20:23:42.182595 3207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260924 20:23:42.182893 3269 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Bootstrap starting.
I20260924 20:23:42.193084 3207 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:46241
I20260924 20:23:42.193325 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:46241 every 8 connection(s)
I20260924 20:23:42.194111 3207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:23:42.198011 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3207
I20260924 20:23:42.198459 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:40971
--local_ip_for_outbound_sockets=127.18.172.66
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=45437
--webserver_interface=127.18.172.66
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:23:42.206338 3269 log.cc:824] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:42.213279 3269 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 20:23:42.213759 3269 tablet_bootstrap.cc:492] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Bootstrap complete.
I20260924 20:23:42.215049 3269 ts_tablet_manager.cc:1397] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20260924 20:23:42.217414 3269 raft_consensus.cc:348] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:42.217852 3269 raft_consensus.cc:729] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb368d712435426683140e8cf60b2260, State: Initialized, Role: FOLLOWER
I20260924 20:23:42.218327 3269 consensus_queue.cc:260] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:42.219329 3269 ts_tablet_manager.cc:1428] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
W20260924 20:23:42.291141 3379 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:23:42.196062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:41673)
0924 20:23:42.196538 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.196591 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:42.197426 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:42.290295 (+ 92869us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":137,"threads_started":1}
W20260924 20:23:42.291785 3377 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:44809 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:44809: Client connection negotiation failed: client connection to 127.18.172.126:44809: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 20:23:42.292551 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.292289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:41637)
0924 20:23:42.292415 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.292485 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 20:23:42.293231 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.293028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:38813)
0924 20:23:42.293132 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.293177 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 20:23:42.293471 3377 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 20:23:42.339223 3380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:42.339495 3380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:42.339581 3380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:42.343794 3380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:42.343861 3380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:42.343925 3380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:23:42.349578 3380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:40971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=45437
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:42.350744 3380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:42.351857 3380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:42.361891 3390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:42.361902 3387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:42.362059 3380 server_base.cc:1034] running on GCE node
W20260924 20:23:42.361902 3388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:42.362547 3380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:42.363214 3380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:42.364477 3380 hybrid_clock.cc:648] HybridClock initialized: now 1790281422364455 us; error 31 us; skew 500 ppm
Sep 24 20:23:42 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281422, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:42.367273 3380 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:23:42.368227 3380 webserver.cc:460] Webserver started at http://127.18.172.66:45437/ using document root <none> and password file <none>
I20260924 20:23:42.368744 3380 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:42.368803 3380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:42.372646 3380 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:23:42.374668 3397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.376011 3380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:23:42.376106 3380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "a20d8a06d6924095837047b55f067e10"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "a92f98d29a8e81c030ded0c5fa107e2f"
server_key_iv: "2fba428f7e629181b77ba3037d561b97"
server_key_version: "encryptionkey@0"
I20260924 20:23:42.376631 3380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:42.404345 3380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:42.407447 3380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:42.407613 3380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:42.408196 3380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:42.408988 3380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:42.409034 3380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.409073 3380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:42.409088 3380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.417995 3380 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:40971
I20260924 20:23:42.418064 3510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:40971 every 8 connection(s)
I20260924 20:23:42.419126 3380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:23:42.423062 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.421266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:55447)
0924 20:23:42.421721 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.421826 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
W20260924 20:23:42.423607 3511 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:44809 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:44809: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
I20260924 20:23:42.424220 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.423967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:53855)
0924 20:23:42.424119 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.424167 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 20:23:42.424969 3513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.424700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:44149)
0924 20:23:42.424826 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.424877 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 20:23:42.425232 3511 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:23:42.428608 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3380
I20260924 20:23:42.429003 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:34803
--local_ip_for_outbound_sockets=127.18.172.67
--tserver_master_addrs=127.18.172.126:44809
--webserver_port=46819
--webserver_interface=127.18.172.67
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:42.543259 3514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:42.543547 3514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:42.543593 3514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:42.547696 3514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:42.547760 3514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:42.547822 3514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:23:42.553120 3514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:34803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=46819
--tserver_master_addrs=127.18.172.126:44809
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:42.554177 3514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:42.555240 3514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:42.565783 3519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:42.565838 3514 server_base.cc:1034] running on GCE node
W20260924 20:23:42.565817 3520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:42.566051 3522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:42.566483 3514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:42.567329 3514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:42.568681 3514 hybrid_clock.cc:648] HybridClock initialized: now 1790281422568673 us; error 34 us; skew 500 ppm
Sep 24 20:23:42 dist-test-slave-m1qq krb5kdc[2058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281422, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:42.571559 3514 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:23:42.572532 3514 webserver.cc:460] Webserver started at http://127.18.172.67:46819/ using document root <none> and password file <none>
I20260924 20:23:42.573117 3514 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:42.573210 3514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:42.577397 3514 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260924 20:23:42.579749 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.581123 3514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:23:42.581266 3514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "8234a807d4004ee0b59ba77b9f3d2883"
format_stamp: "Formatted at 2026-09-24 20:23:00 on dist-test-slave-m1qq"
server_key: "fb5c40bbc3a6643ad574d7ce031cec18"
server_key_iv: "57b900234208298e2885f255d4bbc25b"
server_key_version: "encryptionkey@0"
I20260924 20:23:42.581884 3514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:42.597157 3514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:42.600224 3514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:42.600430 3514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:42.601034 3514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:42.601881 3514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:42.601951 3514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.602029 3514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:42.602073 3514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:42.611390 3514 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:34803
I20260924 20:23:42.611423 3642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:34803 every 8 connection(s)
I20260924 20:23:42.612426 3514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:23:42.615990 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3514
I20260924 20:23:42.616743 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.614678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:44797)
0924 20:23:42.615219 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.615329 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
W20260924 20:23:42.617281 3643 heartbeater.cc:644] Failed to heartbeat to 127.18.172.126:44809 (0 consecutive failures): Network error: Failed to ping master at 127.18.172.126:44809: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
I20260924 20:23:42.617849 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.617649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:36453)
0924 20:23:42.617750 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.617796 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 20:23:42.618393 3645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:42.618220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:45161)
0924 20:23:42.618309 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:42.618347 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 20:23:42.618614 3643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 20:23:43.294595 3647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.294131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:46389)
0924 20:23:43.294408 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:43.294485 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260924 20:23:43.426224 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.425783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:59877)
0924 20:23:43.426038 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:43.426093 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":65,"threads_started":1}
I20260924 20:23:43.619597 3649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.619206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:59667)
0924 20:23:43.619440 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:43.619497 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260924 20:23:43.899343 3650 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:43.899591 3650 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:43.900784 3650 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
Sep 24 20:23:43 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281422, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.67@KRBTEST.COM
Sep 24 20:23:43 dist-test-slave-m1qq krb5kdc[2058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281422, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.66@KRBTEST.COM
I20260924 20:23:43.915138 3652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.901098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:40971 (local address 127.18.172.65:37673)
0924 20:23:43.901706 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:43.901719 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:43.901799 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:43.903182 (+ 1383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:43.903196 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:43.903770 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:43.904275 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:43.904292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.905557 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.905560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:43.906528 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:43.906538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.907303 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.907307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:43.907524 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:43.908250 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:43.908262 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:43.910010 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:43.912598 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:43.912605 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:43.912610 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:43.912853 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:43.913210 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:43.913214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:43.913218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:43.913300 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:43.914131 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:43.914134 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:43.914284 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 20:23:43.914512 (+ 228us) client_negotiation.cc:240] Negotiation successful
0924 20:23:43.914895 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":211,"spinlock_wait_cycles":31104,"thread_start_us":73,"threads_started":1}
I20260924 20:23:43.915184 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.901255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:37673 (local address 127.18.172.66:40971)
0924 20:23:43.901808 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:43.901831 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:43.901893 (+ 62us) server_negotiation.cc:407] Connection header received
0924 20:23:43.902536 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:43.902540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:43.902814 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:43.903122 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:43.904457 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.905415 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.906655 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.907178 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.907353 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:43.910195 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:43.910200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:43.910211 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:43.910237 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:43.912484 (+ 2247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:43.913014 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:43.913018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:43.913021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:43.913120 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:43.913520 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:43.913523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:43.913525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:43.914039 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:43.914130 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:43.914690 (+ 560us) server_negotiation.cc:299] Negotiation successful
0924 20:23:43.914975 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":63,"threads_started":1}
I20260924 20:23:43.915544 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.900974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:34803 (local address 127.18.172.65:54335)
0924 20:23:43.901425 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:43.901442 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:43.901553 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:43.903183 (+ 1630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:43.903189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:43.903770 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:43.904275 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:43.904292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.905559 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.905565 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:43.906528 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:43.906538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.907303 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.907307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:43.907527 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:43.908193 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:43.908210 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:43.909861 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:43.912274 (+ 2413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:43.912280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:43.912294 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:43.912640 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:43.912996 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:43.913000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:43.913005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:43.913149 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:43.913968 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:43.913974 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:43.914284 (+ 310us) client_negotiation.cc:769] Sending connection context
0924 20:23:43.914577 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 20:23:43.915342 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":687,"spinlock_wait_cycles":30720,"thread_start_us":73,"threads_started":1}
I20260924 20:23:43.915839 3653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:43.901169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54335 (local address 127.18.172.67:34803)
0924 20:23:43.901617 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:43.901643 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:43.901688 (+ 45us) server_negotiation.cc:407] Connection header received
0924 20:23:43.902429 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:43.902435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:43.902793 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:43.903122 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:43.904466 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.905411 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.906656 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:43.907179 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:43.907352 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:43.910118 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:43.910123 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:43.910135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:43.910164 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:43.912149 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:43.912739 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:43.912745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:43.912750 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:43.912911 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:43.913266 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:43.913268 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:43.913269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:43.913820 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:43.913980 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:43.914698 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 20:23:43.914985 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":85,"threads_started":1}
W20260924 20:23:43.917745 3266 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
W20260924 20:23:43.917932 3264 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:43.918043 3264 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb368d712435426683140e8cf60b2260; no voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10
I20260924 20:23:43.918318 3650 raft_consensus.cc:2726] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:23:44.295840 3651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:44.295526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:59439)
0924 20:23:44.295687 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:44.295754 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 20:23:44.427719 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:44.427281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:58891)
0924 20:23:44.427574 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:44.427630 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":69,"threads_started":1}
I20260924 20:23:44.620820 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:44.620324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:38653)
0924 20:23:44.620669 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:44.620730 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":71,"threads_started":1}
I20260924 20:23:45.297086 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:45.296670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:44273)
0924 20:23:45.296925 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:45.296988 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":96,"threads_started":1}
I20260924 20:23:45.428831 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:45.428433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:59539)
0924 20:23:45.428686 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:45.428740 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260924 20:23:45.622185 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:45.621730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:40211)
0924 20:23:45.622016 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:45.622084 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":94,"threads_started":1}
I20260924 20:23:45.884101 3660 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:45.884238 3660 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:45.884758 3660 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
W20260924 20:23:45.885665 3266 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
W20260924 20:23:45.885772 3264 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:45.885867 3264 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb368d712435426683140e8cf60b2260; no voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10
I20260924 20:23:45.886080 3660 raft_consensus.cc:2726] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:23:46.298358 3661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:46.297950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:53267)
0924 20:23:46.298210 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:46.298266 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":81,"threads_started":1}
I20260924 20:23:46.430015 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:46.429590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:56657)
0924 20:23:46.429860 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:46.429921 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":86,"threads_started":1}
I20260924 20:23:46.623507 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:46.623077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:44507)
0924 20:23:46.623356 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:46.623417 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":69,"threads_started":1}
I20260924 20:23:47.299801 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:47.299314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:44813)
0924 20:23:47.299641 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:47.299704 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":73,"threads_started":1}
I20260924 20:23:47.431748 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:47.431231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:60015)
0924 20:23:47.431585 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:47.431649 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":71,"threads_started":1}
I20260924 20:23:47.624744 3666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:47.624300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:37715)
0924 20:23:47.624578 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:47.624645 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260924 20:23:48.301045 3667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:48.300602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:50307)
0924 20:23:48.300862 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:48.300948 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260924 20:23:48.433010 3668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:48.432594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:54391)
0924 20:23:48.432857 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:48.432914 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":85,"threads_started":1}
I20260924 20:23:48.625959 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:48.625526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:49949)
0924 20:23:48.625799 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:48.625863 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260924 20:23:48.708123 3670 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:48.708266 3670 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:48.708765 3670 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
W20260924 20:23:48.709599 3264 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
W20260924 20:23:48.709697 3266 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:48.709823 3266 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb368d712435426683140e8cf60b2260; no voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10
I20260924 20:23:48.710048 3670 raft_consensus.cc:2726] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:23:49.302285 3671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:49.301870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:33011)
0924 20:23:49.302131 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:49.302191 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
I20260924 20:23:49.434262 3672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:49.433798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:48559)
0924 20:23:49.434108 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:49.434166 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
I20260924 20:23:49.627621 3673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:49.626949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:34829)
0924 20:23:49.627444 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:49.627509 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":95,"threads_started":1}
I20260924 20:23:50.303673 3674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:50.303234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:47907)
0924 20:23:50.303500 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:50.303568 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260924 20:23:50.435575 3675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:50.435182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:48145)
0924 20:23:50.435423 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:50.435483 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260924 20:23:50.628875 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:50.628387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:52913)
0924 20:23:50.628693 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:50.628746 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":85,"threads_started":1}
I20260924 20:23:51.304862 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:51.304402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:56841)
0924 20:23:51.304675 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:51.304762 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":74,"threads_started":1}
I20260924 20:23:51.436856 3678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:51.436342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:35527)
0924 20:23:51.436696 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:51.436760 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":73,"threads_started":1}
I20260924 20:23:51.630234 3679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:51.629747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:55463)
0924 20:23:51.630045 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:51.630112 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":102,"threads_started":1}
I20260924 20:23:51.648195 3680 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:51.648342 3680 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:51.648842 3680 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
W20260924 20:23:51.649641 3266 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
W20260924 20:23:51.649775 3264 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:51.649868 3264 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb368d712435426683140e8cf60b2260; no voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10
I20260924 20:23:51.650044 3680 raft_consensus.cc:2726] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:23:52.306207 3681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:52.305793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:56107)
0924 20:23:52.306051 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:52.306107 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260924 20:23:52.438076 3682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:52.437661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:50553)
0924 20:23:52.437924 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:52.437981 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260924 20:23:52.631637 3683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:52.631216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:51587)
0924 20:23:52.631477 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:52.631539 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260924 20:23:53.307673 3684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:53.307277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:33253)
0924 20:23:53.307519 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:53.307578 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260924 20:23:53.439312 3685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:53.438862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:53329)
0924 20:23:53.439154 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:53.439214 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":71,"threads_started":1}
I20260924 20:23:53.633018 3686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:53.632560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:53033)
0924 20:23:53.632854 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:53.632919 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":86,"threads_started":1}
I20260924 20:23:54.308847 3687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:54.308422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:37063)
0924 20:23:54.308688 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:54.308752 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260924 20:23:54.440547 3688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:54.440122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:56041)
0924 20:23:54.440389 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:54.440452 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":69,"threads_started":1}
I20260924 20:23:54.634173 3689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:54.633758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:39329)
0924 20:23:54.634017 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:54.634076 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
I20260924 20:23:55.310058 3690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:55.309623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:57765)
0924 20:23:55.309901 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:55.309962 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":75,"threads_started":1}
I20260924 20:23:55.441720 3691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:55.441308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:56169)
0924 20:23:55.441554 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:55.441616 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260924 20:23:55.601960 3692 raft_consensus.cc:482] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:23:55.602154 3692 raft_consensus.cc:504] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8234a807d4004ee0b59ba77b9f3d2883" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 34803 } } peers { permanent_uuid: "fb368d712435426683140e8cf60b2260" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 46241 } } peers { permanent_uuid: "a20d8a06d6924095837047b55f067e10" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 40971 } }
I20260924 20:23:55.602599 3692 leader_election.cc:288] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803), a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971)
W20260924 20:23:55.603556 3264 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8234a807d4004ee0b59ba77b9f3d2883 (127.18.172.67:34803): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
W20260924 20:23:55.603660 3266 leader_election.cc:341] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a20d8a06d6924095837047b55f067e10 (127.18.172.66:40971): Not found: Tablet not found: 5abc6e93516f47668f6c63cd2d5a8734
I20260924 20:23:55.603772 3266 leader_election.cc:302] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb368d712435426683140e8cf60b2260; no voters: 8234a807d4004ee0b59ba77b9f3d2883, a20d8a06d6924095837047b55f067e10
I20260924 20:23:55.603930 3692 raft_consensus.cc:2726] T 5abc6e93516f47668f6c63cd2d5a8734 P fb368d712435426683140e8cf60b2260 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 20:23:55.635686 3693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:55.635213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:59113)
0924 20:23:55.635489 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:55.635548 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260924 20:23:56.311383 3694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:56.310863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:33575)
0924 20:23:56.311213 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:56.311285 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":79,"threads_started":1}
I20260924 20:23:56.443128 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:56.442650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:38475)
0924 20:23:56.442916 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:56.442978 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260924 20:23:56.636831 3696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:56.636435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.67:40539)
0924 20:23:56.636678 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:56.636733 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":78,"threads_started":1}
I20260924 20:23:57.312870 3697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:57.312454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.65:40723)
0924 20:23:57.312709 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:57.312770 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260924 20:23:57.444411 3698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:57.444010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:44809 (local address 127.18.172.66:60925)
0924 20:23:57.444250 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:57.444315 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.126:44809: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260924 20:23:57.624392 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3207
I20260924 20:23:57.630805 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3380
I20260924 20:23:57.636183 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3514
2026-09-24T20:23:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59425 ms)
[----------] 30 tests from SecurityITest (249493 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:23:57 dist-test-slave-m1qq krb5kdc[3701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:23:57 dist-test-slave-m1qq krb5kdc[3701](info): set up 2 sockets
Sep 24 20:23:57 dist-test-slave-m1qq krb5kdc[3701](info): commencing operation
krb5kdc: starting...
W20260924 20:23:58.707983 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:23:58 dist-test-slave-m1qq krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:23:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:23:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:23:58.842026 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34591
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:34591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:58.956691 3714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:58.957036 3714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:58.957139 3714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:58.961321 3714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:23:58.961426 3714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:58.961465 3714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:58.961503 3714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:23:58.961534 3714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:23:58.966960 3714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:34591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:34591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:58.968139 3714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:58.969210 3714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:58.978307 3722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:58.978324 3720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:58.978348 3714 server_base.cc:1034] running on GCE node
W20260924 20:23:58.978370 3719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:58.979298 3714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:58.980456 3714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:58.981650 3714 hybrid_clock.cc:648] HybridClock initialized: now 1790281438981631 us; error 33 us; skew 500 ppm
Sep 24 20:23:58 dist-test-slave-m1qq krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281438, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:58.984511 3714 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:23:58.985404 3714 webserver.cc:460] Webserver started at http://127.18.172.126:40089/ using document root <none> and password file <none>
I20260924 20:23:58.985972 3714 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:58.986068 3714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:58.986304 3714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:58.988379 3714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "9b84b388a5834054a9e94ac15b65845a"
format_stamp: "Formatted at 2026-09-24 20:23:58 on dist-test-slave-m1qq"
server_key: "d646ae92c1d7539574993bd85c2a8568"
server_key_iv: "9ce4a3eddcdb209c0d55b4e058cb4dca"
server_key_version: "encryptionkey@0"
I20260924 20:23:58.988941 3714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "9b84b388a5834054a9e94ac15b65845a"
format_stamp: "Formatted at 2026-09-24 20:23:58 on dist-test-slave-m1qq"
server_key: "d646ae92c1d7539574993bd85c2a8568"
server_key_iv: "9ce4a3eddcdb209c0d55b4e058cb4dca"
server_key_version: "encryptionkey@0"
I20260924 20:23:58.992719 3714 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:23:58.995270 3729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:58.996546 3714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:23:58.996694 3714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "9b84b388a5834054a9e94ac15b65845a"
format_stamp: "Formatted at 2026-09-24 20:23:58 on dist-test-slave-m1qq"
server_key: "d646ae92c1d7539574993bd85c2a8568"
server_key_iv: "9ce4a3eddcdb209c0d55b4e058cb4dca"
server_key_version: "encryptionkey@0"
I20260924 20:23:58.996814 3714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:59.014179 3714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:59.017413 3714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:59.017639 3714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:59.024773 3714 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:34591
I20260924 20:23:59.024793 3781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:34591 every 8 connection(s)
I20260924 20:23:59.025738 3714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:23:59.028590 3782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:59.028648 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3714
I20260924 20:23:59.028846 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:23:59.029268 19121 external_mini_cluster.cc:1428] Setting key fc6c84b8ebfd79bf5eb311f27600af42
I20260924 20:23:59.036302 3782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: Bootstrap starting.
I20260924 20:23:59.038976 3782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: Neither blocks nor log segments found. Creating new log.
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:59.039947 3782 log.cc:824] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:59.042486 3782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: No bootstrap required, opened a new log
I20260924 20:23:59.045802 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.030844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.18.172.126:34591)
0924 20:23:59.031336 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.031356 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.031401 (+ 45us) server_negotiation.cc:407] Connection header received
0924 20:23:59.032160 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.032185 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.032630 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.033090 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:59.033997 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.035542 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.036602 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.037271 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.037432 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.040339 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:59.040348 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:59.040367 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:59.040399 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:59.042467 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.042961 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.042967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.042973 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.043142 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.043467 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.043471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.043472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.044006 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:59.044184 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.044461 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.044817 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":183,"threads_started":1}
I20260924 20:23:59.047194 3782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } }
I20260924 20:23:59.047428 3782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:59.047489 3782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b84b388a5834054a9e94ac15b65845a, State: Initialized, Role: FOLLOWER
I20260924 20:23:59.047920 3782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } }
I20260924 20:23:59.048105 3782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:23:59.048200 3782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:23:59.048300 3782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:23:59.049234 3782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } }
I20260924 20:23:59.049602 3782 leader_election.cc:302] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b84b388a5834054a9e94ac15b65845a; no voters:
I20260924 20:23:59.049911 3782 leader_election.cc:288] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:23:59.050062 3787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:23:59.050318 3787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [term 1 LEADER]: Becoming Leader. State: Replica: 9b84b388a5834054a9e94ac15b65845a, State: Running, Role: LEADER
I20260924 20:23:59.050662 3787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } }
I20260924 20:23:59.051316 3782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:23:59.052417 3789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b84b388a5834054a9e94ac15b65845a. Latest consensus state: current_term: 1 leader_uuid: "9b84b388a5834054a9e94ac15b65845a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } } }
I20260924 20:23:59.052541 3789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:59.052928 3796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:23:59.052985 3788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b84b388a5834054a9e94ac15b65845a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b84b388a5834054a9e94ac15b65845a" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 34591 } } }
I20260924 20:23:59.053134 3788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a [sys.catalog]: This master's current role is: LEADER
I20260924 20:23:59.056152 3796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:23:59.061841 3796 catalog_manager.cc:1347] Generated new cluster ID: 783a387e5c6a4610b3eb61ad4a73a3ff
I20260924 20:23:59.061951 3796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:23:59.068773 3796 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:23:59.069706 3796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:23:59.077327 3796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: Generated new TSK 0
I20260924 20:23:59.078110 3796 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:23:59.119066 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:59.249217 3808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:59.249536 3808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:59.249583 3808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:59.253989 3808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:59.254065 3808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:59.254129 3808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:23:59.259725 3808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:59.260803 3808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:59.261798 3808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:59.272004 3813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.272008 3814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.272011 3816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:59.272519 3808 server_base.cc:1034] running on GCE node
I20260924 20:23:59.272969 3808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:59.273649 3808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:59.274871 3808 hybrid_clock.cc:648] HybridClock initialized: now 1790281439274811 us; error 48 us; skew 500 ppm
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:59.277906 3808 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:23:59.278822 3808 webserver.cc:460] Webserver started at http://127.18.172.65:38835/ using document root <none> and password file <none>
I20260924 20:23:59.279441 3808 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:59.279505 3808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:59.279675 3808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:59.281726 3808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "d8775c20159040f0b130fd0ac3c85edc"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "988265f0b0e6cf6d720ce17861d8946f"
server_key_iv: "a04e85b0b57f3b51ce0b92a7bc9ed8d5"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.282312 3808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "d8775c20159040f0b130fd0ac3c85edc"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "988265f0b0e6cf6d720ce17861d8946f"
server_key_iv: "a04e85b0b57f3b51ce0b92a7bc9ed8d5"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.286202 3808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:23:59.288717 3823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.289968 3808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:23:59.290103 3808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "d8775c20159040f0b130fd0ac3c85edc"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "988265f0b0e6cf6d720ce17861d8946f"
server_key_iv: "a04e85b0b57f3b51ce0b92a7bc9ed8d5"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.290239 3808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:59.304281 3808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:59.307222 3808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:59.307437 3808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:59.308043 3808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:59.308910 3808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:59.308957 3808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.309031 3808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:59.309062 3808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.317714 3808 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:35481
I20260924 20:23:59.317770 3936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:35481 every 8 connection(s)
I20260924 20:23:59.318694 3808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:23:59.326777 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3808
I20260924 20:23:59.326939 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:23:59.327315 19121 external_mini_cluster.cc:1428] Setting key b2a84fda9acce5475826cb524bf2be45
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:59.332551 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.320608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:37199 (local address 127.18.172.126:34591)
0924 20:23:59.320753 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.320757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.321567 (+ 810us) server_negotiation.cc:407] Connection header received
0924 20:23:59.322881 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.322884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.322930 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.323047 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:59.324246 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.324777 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.325874 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.326104 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.326177 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.329026 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:59.329029 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:59.329032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:59.329054 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:59.330479 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.330986 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.330991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.330993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.331130 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.331482 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.331487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.331488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.331670 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:59.331775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.332263 (+ 488us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.332388 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:23:59.333454 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.320973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34591 (local address 127.18.172.65:37199)
0924 20:23:59.321408 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.321451 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.322675 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.323238 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.323245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.323605 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:59.324103 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.324117 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.324894 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.324896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.325767 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.325776 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.326235 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.326238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.326388 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.327118 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:59.327141 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:59.328863 (+ 1722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:59.330606 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.330612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.330628 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.330887 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.331234 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.331237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.331239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.331390 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.331775 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:59.331784 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:59.332037 (+ 253us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.332225 (+ 188us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.332576 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260924 20:23:59.334947 3937 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34591
I20260924 20:23:59.335273 3937 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:59.335897 3937 heartbeater.cc:511] Master 127.18.172.126:34591 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:23:59.337811 3746 ts_manager.cc:194] Registered new tserver with Master: d8775c20159040f0b130fd0ac3c85edc (127.18.172.65:35481)
I20260924 20:23:59.339072 3746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:37199
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:23:59.358803 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:59.470436 3942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:59.470700 3942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:59.470744 3942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:59.474949 3942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:59.475070 3942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:59.475183 3942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:23:59.480651 3942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:59.481751 3942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:59.482764 3942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:23:59.494043 3947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.494048 3948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.494069 3950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:59.494316 3942 server_base.cc:1034] running on GCE node
I20260924 20:23:59.494845 3942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:59.495589 3942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:59.496749 3942 hybrid_clock.cc:648] HybridClock initialized: now 1790281439496714 us; error 56 us; skew 500 ppm
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:59.499490 3942 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:23:59.500458 3942 webserver.cc:460] Webserver started at http://127.18.172.66:45461/ using document root <none> and password file <none>
I20260924 20:23:59.501080 3942 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:59.501173 3942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:59.501394 3942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:59.503490 3942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "02256b61d72e4499ae176a0768163242"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "3ac43a701b370308cbc60cd0b82df355"
server_key_iv: "2ae61d01532cb8bde0f4b8cf7a9c69f0"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.504006 3942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "02256b61d72e4499ae176a0768163242"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "3ac43a701b370308cbc60cd0b82df355"
server_key_iv: "2ae61d01532cb8bde0f4b8cf7a9c69f0"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.507632 3942 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:23:59.510039 3957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.511336 3942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:23:59.511464 3942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "02256b61d72e4499ae176a0768163242"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "3ac43a701b370308cbc60cd0b82df355"
server_key_iv: "2ae61d01532cb8bde0f4b8cf7a9c69f0"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.511574 3942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:59.540133 3942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:59.543119 3942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:59.543332 3942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:59.543937 3942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:59.544818 3942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:59.544899 3942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.544961 3942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:59.544997 3942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.554052 3942 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:34923
I20260924 20:23:59.554073 4070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:34923 every 8 connection(s)
I20260924 20:23:59.554976 3942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:23:59.555667 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 3942
I20260924 20:23:59.555792 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:23:59.556109 19121 external_mini_cluster.cc:1428] Setting key 10ee105a311d2922e1ec26fa9207d97f
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:59.574412 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.557119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:35091 (local address 127.18.172.126:34591)
0924 20:23:59.557257 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.557261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.558272 (+ 1011us) server_negotiation.cc:407] Connection header received
0924 20:23:59.560239 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.560258 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.560357 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.560475 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:59.561665 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.562579 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.563798 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.564201 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.564316 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.570015 (+ 5699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:59.570018 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:59.570020 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:59.570054 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:59.571910 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.572626 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.572630 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.572631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.572720 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.573150 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.573153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.573154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.573347 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:59.573427 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.574167 (+ 740us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.574284 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 20:23:59.575806 4073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.557603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34591 (local address 127.18.172.66:35091)
0924 20:23:59.558133 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.558176 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.559866 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.560675 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.560683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.561025 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:59.561517 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.561531 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.562702 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.562704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.563671 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.563680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.564355 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.564358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.564509 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.565135 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:59.565152 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:59.569701 (+ 4549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:59.572060 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.572070 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.572092 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.572494 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.572821 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.572827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.572832 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.573047 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.573462 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:59.573472 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:59.573880 (+ 408us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.574107 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.574517 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":152,"threads_started":1}
I20260924 20:23:59.577173 4071 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34591
I20260924 20:23:59.577471 4071 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:59.578027 4071 heartbeater.cc:511] Master 127.18.172.126:34591 requested a full tablet report, sending...
I20260924 20:23:59.579175 3746 ts_manager.cc:194] Registered new tserver with Master: 02256b61d72e4499ae176a0768163242 (127.18.172.66:34923)
I20260924 20:23:59.579772 3746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:35091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:23:59.586877 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:45845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:23:59.700505 4076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:23:59.700835 4076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:23:59.700917 4076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:23:59.705092 4076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:23:59.705150 4076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:23:59.705257 4076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:23:59.710701 4076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:45845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:34591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:23:59.711829 4076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:23:59.712992 4076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:23:59.723093 4076 server_base.cc:1034] running on GCE node
W20260924 20:23:59.723067 4082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.723066 4081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:23:59.723112 4084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:23:59.723726 4076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:23:59.724480 4076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:23:59.725680 4076 hybrid_clock.cc:648] HybridClock initialized: now 1790281439725630 us; error 80 us; skew 500 ppm
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:23:59.728617 4076 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:23:59.729517 4076 webserver.cc:460] Webserver started at http://127.18.172.67:34053/ using document root <none> and password file <none>
I20260924 20:23:59.730099 4076 fs_manager.cc:359] Metadata directory not provided
I20260924 20:23:59.730192 4076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:23:59.730430 4076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:23:59.732570 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "479ddaa867cb4209946ba0a2048a8e0a"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "9e2a66179ea6f1279966ae06cc0e185d"
server_key_iv: "6d171e6e1b307c80b3445b67b017386b"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.733142 4076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "479ddaa867cb4209946ba0a2048a8e0a"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "9e2a66179ea6f1279966ae06cc0e185d"
server_key_iv: "6d171e6e1b307c80b3445b67b017386b"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.737023 4076 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:23:59.739634 4091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.740923 4076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:23:59.741066 4076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "479ddaa867cb4209946ba0a2048a8e0a"
format_stamp: "Formatted at 2026-09-24 20:23:59 on dist-test-slave-m1qq"
server_key: "9e2a66179ea6f1279966ae06cc0e185d"
server_key_iv: "6d171e6e1b307c80b3445b67b017386b"
server_key_version: "encryptionkey@0"
I20260924 20:23:59.741186 4076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:23:59.766408 4076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:23:59.769999 4076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:23:59.770218 4076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:23:59.770859 4076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:23:59.771740 4076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:23:59.771814 4076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.771888 4076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:23:59.771932 4076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:23:59.781100 4076 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:45045
I20260924 20:23:59.781131 4204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:45045 every 8 connection(s)
I20260924 20:23:59.782017 4076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:23:59.783483 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4076
I20260924 20:23:59.783847 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:23:59.784291 19121 external_mini_cluster.cc:1428] Setting key b4004c3db48cdb0db34c842ce6243277
Sep 24 20:23:59 dist-test-slave-m1qq krb5kdc[3701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281439, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:23:59.796300 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.784135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:36079 (local address 127.18.172.126:34591)
0924 20:23:59.784296 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.784302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.785332 (+ 1030us) server_negotiation.cc:407] Connection header received
0924 20:23:59.786965 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.786967 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.787058 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.787142 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:59.788414 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.788938 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.790122 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.790329 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.790407 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.792810 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:59.792813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:59.792815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:59.792839 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:59.794363 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.794917 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.794920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.794921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.794993 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.795337 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.795340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.795341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.795521 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:59.795597 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.796081 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.796177 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:23:59.797643 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.784574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:34591 (local address 127.18.172.67:36079)
0924 20:23:59.785147 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.785216 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.786757 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.787378 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.787385 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.787753 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:23:59.788283 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.788298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.789068 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.789071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.790013 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.790023 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.790453 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.790455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.790592 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.791184 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:23:59.791200 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:23:59.792660 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:23:59.794507 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.794513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.794526 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.794822 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.795118 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:23:59.795121 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:23:59.795122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:23:59.795256 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:23:59.795610 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:23:59.795615 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:23:59.795869 (+ 254us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.796035 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.796284 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260924 20:23:59.798959 4205 heartbeater.cc:347] Connected to a master server at 127.18.172.126:34591
I20260924 20:23:59.799322 4205 heartbeater.cc:464] Registering TS with master...
I20260924 20:23:59.799989 4205 heartbeater.cc:511] Master 127.18.172.126:34591 requested a full tablet report, sending...
I20260924 20:23:59.801121 3746 ts_manager.cc:194] Registered new tserver with Master: 479ddaa867cb4209946ba0a2048a8e0a (127.18.172.67:45045)
I20260924 20:23:59.801753 3746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:36079
I20260924 20:23:59.810467 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:23:59.819854 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.812466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43562 (local address 127.18.172.126:34591)
0924 20:23:59.812640 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.812644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.812719 (+ 75us) server_negotiation.cc:407] Connection header received
0924 20:23:59.812830 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.812832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.812875 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.812940 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:23:59.813607 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.814118 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.815263 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.815505 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.815602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.816959 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:23:59.816962 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:23:59.816964 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:23:59.816988 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:23:59.818255 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.818642 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.818645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.818646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.818721 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:23:59.818947 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:23:59.818949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:23:59.818950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:23:59.819182 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:23:59.819288 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.819522 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.819675 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 20:23:59.823153 3746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:23:59.826301 3746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:23:59.840693 4216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.834887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:35481 (local address 127.0.0.1:44748)
0924 20:23:59.835282 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.835338 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.835476 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.836060 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.836067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.836092 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:59.836246 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.836256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.837811 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.837814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.838941 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.838950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.840123 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.840129 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.840245 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.840289 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.840414 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.840537 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":113,"threads_started":1}
I20260924 20:23:59.840703 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.835076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.18.172.65:35481)
0924 20:23:59.835519 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.835528 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.835559 (+ 31us) server_negotiation.cc:407] Connection header received
0924 20:23:59.835639 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.835646 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.835873 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.836425 (+ 552us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:59.836578 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.837692 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.839103 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.839999 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.840112 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.840257 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.840448 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.840548 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":125,"threads_started":1}
I20260924 20:23:59.841470 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.835137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:45045 (local address 127.0.0.1:56652)
0924 20:23:59.835708 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.835720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.835790 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.837021 (+ 1231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.837025 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.837044 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:59.837148 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.837153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.838451 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.838456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.839819 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.839830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.841058 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.841061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.841157 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.841184 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.841261 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.841339 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":92,"thread_start_us":82,"threads_started":1}
I20260924 20:23:59.841701 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.835744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.18.172.67:45045)
0924 20:23:59.836103 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.836121 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.836161 (+ 40us) server_negotiation.cc:407] Connection header received
0924 20:23:59.836239 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.836244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.836877 (+ 633us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.837032 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:59.837266 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.838292 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.839947 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.840759 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.840874 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.841005 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.841316 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.841434 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":181,"threads_started":1}
I20260924 20:23:59.841713 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.835677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34923 (local address 127.0.0.1:50050)
0924 20:23:59.836420 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0924 20:23:59.836432 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:23:59.836520 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:23:59.837264 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:23:59.837267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:23:59.837286 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:23:59.837395 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.837402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.838514 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.838517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.840021 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:23:59.840029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.841320 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.841325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:23:59.841431 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.841453 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:23:59.841526 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:23:59.841591 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":322,"thread_start_us":102,"threads_started":1}
I20260924 20:23:59.842139 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:23:59.836256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.18.172.66:34923)
0924 20:23:59.836670 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0924 20:23:59.836684 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:23:59.836705 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:23:59.836778 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:23:59.836783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:23:59.836950 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:23:59.837081 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:23:59.837519 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.838345 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.840167 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:23:59.840995 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:23:59.841090 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:23:59.841256 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 20:23:59.841901 (+ 645us) server_negotiation.cc:299] Negotiation successful
0924 20:23:59.842006 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":133,"threads_started":1}
I20260924 20:23:59.844135 3871 tablet_service.cc:1467] Processing CreateTablet for tablet c56d106e413a4e009131fd2af4d0d3ee (DEFAULT_TABLE table=test-table [id=6ce8ce5f96fb42fc9dfe4d42a2fee6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:59.844547 4005 tablet_service.cc:1467] Processing CreateTablet for tablet c56d106e413a4e009131fd2af4d0d3ee (DEFAULT_TABLE table=test-table [id=6ce8ce5f96fb42fc9dfe4d42a2fee6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:59.844547 4139 tablet_service.cc:1467] Processing CreateTablet for tablet c56d106e413a4e009131fd2af4d0d3ee (DEFAULT_TABLE table=test-table [id=6ce8ce5f96fb42fc9dfe4d42a2fee6de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:23:59.845153 3871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56d106e413a4e009131fd2af4d0d3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:59.845515 4139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56d106e413a4e009131fd2af4d0d3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:59.845515 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56d106e413a4e009131fd2af4d0d3ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:23:59.852294 4222 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Bootstrap starting.
I20260924 20:23:59.853053 4224 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Bootstrap starting.
I20260924 20:23:59.853243 4223 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Bootstrap starting.
I20260924 20:23:59.854540 4222 tablet_bootstrap.cc:654] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:59.855357 4222 log.cc:824] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:59.855433 4223 tablet_bootstrap.cc:654] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:59.855433 4224 tablet_bootstrap.cc:654] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Neither blocks nor log segments found. Creating new log.
I20260924 20:23:59.856824 4224 log.cc:824] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:59.856824 4223 log.cc:824] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Log is configured to *not* fsync() on all Append() calls
I20260924 20:23:59.857381 4222 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: No bootstrap required, opened a new log
I20260924 20:23:59.857596 4222 ts_tablet_manager.cc:1397] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:23:59.858767 4224 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: No bootstrap required, opened a new log
I20260924 20:23:59.858973 4224 ts_tablet_manager.cc:1397] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:23:59.859234 4223 tablet_bootstrap.cc:492] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: No bootstrap required, opened a new log
I20260924 20:23:59.859433 4223 ts_tablet_manager.cc:1397] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:23:59.861838 4224 raft_consensus.cc:348] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.862097 4224 raft_consensus.cc:374] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:59.861963 4222 raft_consensus.cc:348] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.862169 4224 raft_consensus.cc:729] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479ddaa867cb4209946ba0a2048a8e0a, State: Initialized, Role: FOLLOWER
I20260924 20:23:59.862241 4222 raft_consensus.cc:374] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:59.862298 4222 raft_consensus.cc:729] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8775c20159040f0b130fd0ac3c85edc, State: Initialized, Role: FOLLOWER
I20260924 20:23:59.862654 4224 consensus_queue.cc:260] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.862722 4222 consensus_queue.cc:260] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.863312 4205 heartbeater.cc:503] Master 127.18.172.126:34591 was elected leader, sending a full tablet report...
I20260924 20:23:59.863189 4223 raft_consensus.cc:348] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.863410 4223 raft_consensus.cc:374] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:23:59.863461 4223 raft_consensus.cc:729] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02256b61d72e4499ae176a0768163242, State: Initialized, Role: FOLLOWER
I20260924 20:23:59.863812 4222 ts_tablet_manager.cc:1428] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:23:59.863880 4223 consensus_queue.cc:260] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:23:59.864135 3937 heartbeater.cc:503] Master 127.18.172.126:34591 was elected leader, sending a full tablet report...
I20260924 20:23:59.864663 4224 ts_tablet_manager.cc:1428] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:23:59.864699 4223 ts_tablet_manager.cc:1428] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:23:59.865156 4071 heartbeater.cc:503] Master 127.18.172.126:34591 was elected leader, sending a full tablet report...
W20260924 20:24:00.033022 4206 tablet.cc:2369] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:00.033581 4229 raft_consensus.cc:482] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:00.033775 4229 raft_consensus.cc:504] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:24:00.035471 4229 leader_election.cc:288] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02256b61d72e4499ae176a0768163242 (127.18.172.66:34923), 479ddaa867cb4209946ba0a2048a8e0a (127.18.172.67:45045)
I20260924 20:24:00.040421 4220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.035863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:46663 (local address 127.18.172.67:45045)
0924 20:24:00.036020 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.036027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.036279 (+ 252us) server_negotiation.cc:407] Connection header received
0924 20:24:00.036468 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.036471 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.036522 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.036599 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:00.036822 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.037326 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.038986 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.039615 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.039742 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.039794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.040214 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.040285 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 20:24:00.040421 4221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.035695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:60805 (local address 127.18.172.66:34923)
0924 20:24:00.035824 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.035829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.036130 (+ 301us) server_negotiation.cc:407] Connection header received
0924 20:24:00.036275 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.036278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.036340 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.036421 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:00.036786 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.037540 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.038980 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.039613 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.039742 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.039794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.040214 (+ 420us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.040285 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:24:00.040580 4232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.035441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:45045 (local address 127.18.172.65:46663)
0924 20:24:00.036195 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.036207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:00.036341 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:00.036607 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:00.036610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:00.036622 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:00.036727 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.036736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.037453 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.037457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.038838 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.038852 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.039743 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.039747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.040007 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.040102 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 20:24:00.040181 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:24:00.040357 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":543,"thread_start_us":120,"threads_started":1}
I20260924 20:24:00.040580 4231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.035441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34923 (local address 127.18.172.65:60805)
0924 20:24:00.036030 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.036044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:00.036169 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:00.036466 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:00.036468 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:00.036483 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:00.036624 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.036631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.037668 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.037671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.038840 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.038852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.039743 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.039747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.040004 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.040104 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 20:24:00.040181 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 20:24:00.040357 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":203,"thread_start_us":126,"threads_started":1}
I20260924 20:24:00.041349 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56d106e413a4e009131fd2af4d0d3ee" candidate_uuid: "d8775c20159040f0b130fd0ac3c85edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02256b61d72e4499ae176a0768163242" is_pre_election: true
I20260924 20:24:00.041294 4159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56d106e413a4e009131fd2af4d0d3ee" candidate_uuid: "d8775c20159040f0b130fd0ac3c85edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479ddaa867cb4209946ba0a2048a8e0a" is_pre_election: true
I20260924 20:24:00.041693 4159 raft_consensus.cc:2454] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8775c20159040f0b130fd0ac3c85edc in term 0.
I20260924 20:24:00.041693 4025 raft_consensus.cc:2454] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8775c20159040f0b130fd0ac3c85edc in term 0.
I20260924 20:24:00.042212 3826 leader_election.cc:302] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02256b61d72e4499ae176a0768163242, 479ddaa867cb4209946ba0a2048a8e0a, d8775c20159040f0b130fd0ac3c85edc; no voters:
I20260924 20:24:00.042534 4229 raft_consensus.cc:2781] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:00.042642 4229 raft_consensus.cc:482] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:00.042725 4229 raft_consensus.cc:3037] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:00.043850 4229 raft_consensus.cc:504] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:24:00.044294 4229 leader_election.cc:288] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [CANDIDATE]: Term 1 election: Requested vote from peers 02256b61d72e4499ae176a0768163242 (127.18.172.66:34923), 479ddaa867cb4209946ba0a2048a8e0a (127.18.172.67:45045)
I20260924 20:24:00.044670 4025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56d106e413a4e009131fd2af4d0d3ee" candidate_uuid: "d8775c20159040f0b130fd0ac3c85edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02256b61d72e4499ae176a0768163242"
I20260924 20:24:00.044677 4159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56d106e413a4e009131fd2af4d0d3ee" candidate_uuid: "d8775c20159040f0b130fd0ac3c85edc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479ddaa867cb4209946ba0a2048a8e0a"
I20260924 20:24:00.044802 4159 raft_consensus.cc:3037] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:00.044827 4025 raft_consensus.cc:3037] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:00.046159 4025 raft_consensus.cc:2454] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8775c20159040f0b130fd0ac3c85edc in term 1.
I20260924 20:24:00.046159 4159 raft_consensus.cc:2454] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8775c20159040f0b130fd0ac3c85edc in term 1.
I20260924 20:24:00.046522 3825 leader_election.cc:302] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02256b61d72e4499ae176a0768163242, 479ddaa867cb4209946ba0a2048a8e0a, d8775c20159040f0b130fd0ac3c85edc; no voters:
I20260924 20:24:00.046718 4229 raft_consensus.cc:2781] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:00.046977 4229 raft_consensus.cc:686] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 1 LEADER]: Becoming Leader. State: Replica: d8775c20159040f0b130fd0ac3c85edc, State: Running, Role: LEADER
I20260924 20:24:00.047353 4229 consensus_queue.cc:237] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } }
I20260924 20:24:00.050457 3744 catalog_manager.cc:5495] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc reported cstate change: term changed from 0 to 1, leader changed from <none> to d8775c20159040f0b130fd0ac3c85edc (127.18.172.65). New cstate: current_term: 1 leader_uuid: "d8775c20159040f0b130fd0ac3c85edc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8775c20159040f0b130fd0ac3c85edc" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 } health_report { overall_health: UNKNOWN } } }
W20260924 20:24:00.056200 4072 tablet.cc:2369] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 20:24:00.070065 3938 tablet.cc:2369] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:00.076684 4217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.073336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.18.172.65:35481)
0924 20:24:00.073480 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.073484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.073498 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:24:00.073611 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.073614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.073667 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.073752 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:00.073975 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.074475 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.075325 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.075557 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.075636 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.075890 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:00.076294 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:00.076413 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.076458 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.076506 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:24:00.082808 4159 raft_consensus.cc:1261] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 1 FOLLOWER]: Refusing update from remote peer d8775c20159040f0b130fd0ac3c85edc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:00.082806 4025 raft_consensus.cc:1261] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 1 FOLLOWER]: Refusing update from remote peer d8775c20159040f0b130fd0ac3c85edc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:00.083626 4229 consensus_queue.cc:1035] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [LEADER]: Connected to new peer: Peer: permanent_uuid: "02256b61d72e4499ae176a0768163242" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 34923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:00.083840 4233 consensus_queue.cc:1035] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [LEADER]: Connected to new peer: Peer: permanent_uuid: "479ddaa867cb4209946ba0a2048a8e0a" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:00.092178 4237 mvcc.cc:205] Tried to move back new op lower bound from 7332992778571337728 to 7332992778436489216. Current Snapshot: MvccSnapshot[applied={T|T < 7332992778571337728}]
I20260924 20:24:00.099874 3745 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:00.100063 3745 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:00.102569 3745 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b84b388a5834054a9e94ac15b65845a: Sending DeleteTablet for 3 replicas of tablet c56d106e413a4e009131fd2af4d0d3ee
I20260924 20:24:00.103346 3871 tablet_service.cc:1514] Processing DeleteTablet for tablet c56d106e413a4e009131fd2af4d0d3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:44748
I20260924 20:24:00.103520 4139 tablet_service.cc:1514] Processing DeleteTablet for tablet c56d106e413a4e009131fd2af4d0d3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:56652
I20260924 20:24:00.103991 4247 tablet_replica.cc:330] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: stopping tablet replica
I20260924 20:24:00.104175 4246 tablet_replica.cc:330] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc: stopping tablet replica
I20260924 20:24:00.104262 4005 tablet_service.cc:1514] Processing DeleteTablet for tablet c56d106e413a4e009131fd2af4d0d3ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:00 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:50050
I20260924 20:24:00.104387 4247 raft_consensus.cc:2229] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:00.104635 4246 raft_consensus.cc:2229] T c56d106e413a4e009131fd2af4d0d3ee P d8775c20159040f0b130fd0ac3c85edc [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:24:00.104754 4247 raft_consensus.cc:2258] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:00.105077 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3808
I20260924 20:24:00.107653 4247 ts_tablet_manager.cc:1905] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:00.111152 4248 tablet_replica.cc:330] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: stopping tablet replica
I20260924 20:24:00.111248 4247 ts_tablet_manager.cc:1918] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:00.111357 4247 log.cc:1196] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/c56d106e413a4e009131fd2af4d0d3ee
I20260924 20:24:00.111517 4248 raft_consensus.cc:2229] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:00.111759 4247 ts_tablet_manager.cc:1939] T c56d106e413a4e009131fd2af4d0d3ee P 479ddaa867cb4209946ba0a2048a8e0a: Deleting consensus metadata
I20260924 20:24:00.111922 4248 raft_consensus.cc:2258] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 20:24:00.113138 3732 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:35481 (error 108)
I20260924 20:24:00.113184 3731 catalog_manager.cc:4841] TS 479ddaa867cb4209946ba0a2048a8e0a (127.18.172.67:45045): tablet c56d106e413a4e009131fd2af4d0d3ee (table test-table [id=6ce8ce5f96fb42fc9dfe4d42a2fee6de]) successfully deleted
I20260924 20:24:00.113499 4248 ts_tablet_manager.cc:1905] T c56d106e413a4e009131fd2af4d0d3ee P 02256b61d72e4499ae176a0768163242: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:00.113979 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3942
I20260924 20:24:00.114684 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.114312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:35481 (local address 127.0.0.1:44752)
0924 20:24:00.114500 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.114578 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:35481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 20:24:00.114882 3732 catalog_manager.cc:4584] TS d8775c20159040f0b130fd0ac3c85edc (127.18.172.65:35481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c56d106e413a4e009131fd2af4d0d3ee: Network error: Client connection negotiation failed: client connection to 127.18.172.65:35481: connect: Connection refused (error 111)
I20260924 20:24:00.122576 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4076
I20260924 20:24:00.124480 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.122837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:34923 (local address 127.0.0.1:50060)
0924 20:24:00.124296 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.124368 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:34923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1347}
I20260924 20:24:00.128129 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 3714
2026-09-24T20:24:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2491 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T20:24:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:00Z Disabled control of system clock
I20260924 20:24:00.159124 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41613
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:00.273406 4253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:00.273778 4253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:00.273823 4253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:00.277928 4253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:00.277993 4253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:00.278009 4253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:00.278028 4253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:00.278043 4253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:00.282829 4253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:41613
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:00.284045 4253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:00.285374 4253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:00.294103 4258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.294157 4261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.294103 4259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.294256 4253 server_base.cc:1034] running on GCE node
I20260924 20:24:00.295079 4253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:00.296183 4253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:00.297369 4253 hybrid_clock.cc:648] HybridClock initialized: now 1790281440297342 us; error 26 us; skew 500 ppm
I20260924 20:24:00.310527 4253 webserver.cc:460] Webserver started at http://127.18.172.126:44169/ using document root <none> and password file <none>
I20260924 20:24:00.311205 4253 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:00.311280 4253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:00.311466 4253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:00.313591 4253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "c67d2ad5f1b542e8a0921ae96ad13741"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "67314983335f231fc31444fa85f53278"
server_key_iv: "60f9f6618405c3ad82bb3669d13bd576"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.314100 4253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "c67d2ad5f1b542e8a0921ae96ad13741"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "67314983335f231fc31444fa85f53278"
server_key_iv: "60f9f6618405c3ad82bb3669d13bd576"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.317727 4253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 20:24:00.320278 4267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.321543 4253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:00.321645 4253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "c67d2ad5f1b542e8a0921ae96ad13741"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "67314983335f231fc31444fa85f53278"
server_key_iv: "60f9f6618405c3ad82bb3669d13bd576"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.321736 4253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:00.347105 4253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:00.347806 4253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:00.347960 4253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:00.355440 4253 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:41613
I20260924 20:24:00.355456 4319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:41613 every 8 connection(s)
I20260924 20:24:00.356484 4253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:00.359730 4320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:00.365353 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4253
I20260924 20:24:00.365515 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:00.365825 19121 external_mini_cluster.cc:1428] Setting key 4d1b63a919750935e93e6ed0afdf1852
I20260924 20:24:00.365942 4320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: Bootstrap starting.
I20260924 20:24:00.368849 4320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:00.369668 4320 log.cc:824] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:00.371981 4320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: No bootstrap required, opened a new log
I20260924 20:24:00.375826 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.367276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.18.172.126:41613)
0924 20:24:00.368058 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.368074 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.368111 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:24:00.368797 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.368815 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.369295 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.369650 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:00.370143 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.371346 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.372305 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.372793 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.372980 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.373928 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:00.373933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:00.373947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:00.374348 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:00.374407 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:00.374427 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:00.374540 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.374695 (+ 155us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.374987 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":172,"threads_started":1}
I20260924 20:24:00.376174 4320 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } }
I20260924 20:24:00.376483 4320 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:00.376534 4320 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67d2ad5f1b542e8a0921ae96ad13741, State: Initialized, Role: FOLLOWER
I20260924 20:24:00.377010 4320 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } }
I20260924 20:24:00.377142 4320 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:00.377225 4320 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:00.377357 4320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:00.378509 4320 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } }
I20260924 20:24:00.378860 4320 leader_election.cc:302] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67d2ad5f1b542e8a0921ae96ad13741; no voters:
I20260924 20:24:00.379201 4320 leader_election.cc:288] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:00.379338 4325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:00.379572 4325 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [term 1 LEADER]: Becoming Leader. State: Replica: c67d2ad5f1b542e8a0921ae96ad13741, State: Running, Role: LEADER
I20260924 20:24:00.379918 4325 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } }
I20260924 20:24:00.380507 4320 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:00.381719 4326 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } } }
I20260924 20:24:00.381876 4326 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:00.381717 4327 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67d2ad5f1b542e8a0921ae96ad13741. Latest consensus state: current_term: 1 leader_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67d2ad5f1b542e8a0921ae96ad13741" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 41613 } } }
I20260924 20:24:00.381973 4327 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:00.382371 4334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:00.385463 4334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:00.391301 4334 catalog_manager.cc:1347] Generated new cluster ID: f3f13ee52250476799fe524d20775a04
I20260924 20:24:00.391402 4334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:00.405382 4334 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:00.406348 4334 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:00.419993 4334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: Generated new TSK 0
I20260924 20:24:00.420909 4334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:00.431530 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--builtin_ntp_servers=127.18.172.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:00.548622 4344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:00.548944 4344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:00.549023 4344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:00.553184 4344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:00.553498 4344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:00.553646 4344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:24:00.558670 4344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:00.559808 4344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:00.561203 4344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:00.570995 4349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.570995 4350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.571026 4352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.571460 4344 server_base.cc:1034] running on GCE node
I20260924 20:24:00.571830 4344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:00.572533 4344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:00.573768 4344 hybrid_clock.cc:648] HybridClock initialized: now 1790281440573746 us; error 31 us; skew 500 ppm
I20260924 20:24:00.575781 4344 webserver.cc:460] Webserver started at http://127.18.172.65:46059/ using document root <none> and password file <none>
I20260924 20:24:00.576390 4344 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:00.576453 4344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:00.576704 4344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:00.578835 4344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "626e8a76920b4a0a8b33ce67936000c2"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "260a5ab1fc425f68c54b11c838d124e6"
server_key_iv: "10c2d25e3376c9d334732159f7799f0e"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.579484 4344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "626e8a76920b4a0a8b33ce67936000c2"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "260a5ab1fc425f68c54b11c838d124e6"
server_key_iv: "10c2d25e3376c9d334732159f7799f0e"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.583425 4344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:24:00.585830 4358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.587132 4344 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:00.587246 4344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "626e8a76920b4a0a8b33ce67936000c2"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "260a5ab1fc425f68c54b11c838d124e6"
server_key_iv: "10c2d25e3376c9d334732159f7799f0e"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.587355 4344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:00.607802 4344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:00.608510 4344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:00.608707 4344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:00.609302 4344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:00.610250 4344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:00.610294 4344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.610359 4344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:00.610395 4344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.619113 4344 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:35797
I20260924 20:24:00.619156 4471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:35797 every 8 connection(s)
I20260924 20:24:00.619990 4344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:24:00.629056 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4344
I20260924 20:24:00.629220 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:24:00.629593 19121 external_mini_cluster.cc:1428] Setting key 0c20709bd6687542ef613be212fb0ecc
I20260924 20:24:00.629731 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.621796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:42817 (local address 127.18.172.126:41613)
0924 20:24:00.621964 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.621969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.623026 (+ 1057us) server_negotiation.cc:407] Connection header received
0924 20:24:00.624505 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.624508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.624570 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.624677 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:00.625707 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.626285 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.627659 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.627906 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.628022 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.629155 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:00.629158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:00.629161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:00.629200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:00.629227 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:00.629239 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:00.629306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.629505 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.629571 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:24:00.630700 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.622199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41613 (local address 127.18.172.65:42817)
0924 20:24:00.622849 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.622892 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:00.624262 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:00.624874 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:00.624880 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:00.624997 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:00.625570 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.625585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.626432 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.626434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.627519 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.627535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.628067 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.628071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.628237 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.628884 (+ 647us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:00.628886 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:00.628928 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:00.628993 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:00.629307 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:00.629319 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:00.629327 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:24:00.629475 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 20:24:00.629750 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":111,"threads_started":1}
I20260924 20:24:00.631675 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--builtin_ntp_servers=127.18.172.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:00.632033 4472 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41613
I20260924 20:24:00.632305 4472 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:00.632853 4472 heartbeater.cc:511] Master 127.18.172.126:41613 requested a full tablet report, sending...
I20260924 20:24:00.634326 4284 ts_manager.cc:194] Registered new tserver with Master: 626e8a76920b4a0a8b33ce67936000c2 (127.18.172.65:35797)
I20260924 20:24:00.635524 4284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.65:42817
W20260924 20:24:00.743641 4475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:00.743968 4475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:00.744038 4475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:00.748286 4475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:00.748368 4475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:00.748454 4475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:24:00.753422 4475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:00.754582 4475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:00.756135 4475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:00.766142 4480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.766213 4475 server_base.cc:1034] running on GCE node
W20260924 20:24:00.766168 4481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.766168 4483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.766860 4475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:00.767570 4475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:00.768814 4475 hybrid_clock.cc:648] HybridClock initialized: now 1790281440768796 us; error 32 us; skew 500 ppm
I20260924 20:24:00.770874 4475 webserver.cc:460] Webserver started at http://127.18.172.66:42099/ using document root <none> and password file <none>
I20260924 20:24:00.771543 4475 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:00.771638 4475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:00.771864 4475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:00.774029 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "e9a6a30d0377499380ee4a819418b106"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "f5a92567780a69ec973461f3d2f98a48"
server_key_iv: "fbc68d33fbd1cfbfa6c0c0ad09299c01"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.774605 4475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "e9a6a30d0377499380ee4a819418b106"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "f5a92567780a69ec973461f3d2f98a48"
server_key_iv: "fbc68d33fbd1cfbfa6c0c0ad09299c01"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.778466 4475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:24:00.781252 4489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.782755 4475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:00.782888 4475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "e9a6a30d0377499380ee4a819418b106"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "f5a92567780a69ec973461f3d2f98a48"
server_key_iv: "fbc68d33fbd1cfbfa6c0c0ad09299c01"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.783044 4475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:00.795248 4475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:00.795943 4475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:00.796141 4475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:00.796720 4475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:00.797684 4475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:00.797755 4475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.797827 4475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:00.797883 4475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.806772 4475 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:38321
I20260924 20:24:00.806802 4602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:38321 every 8 connection(s)
I20260924 20:24:00.807806 4475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:24:00.808938 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4475
I20260924 20:24:00.809041 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:24:00.809444 19121 external_mini_cluster.cc:1428] Setting key df830f4d522043c6bd1e4bd9f8d3a062
I20260924 20:24:00.811704 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--builtin_ntp_servers=127.18.172.84:34959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:00.819593 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.809851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:47901 (local address 127.18.172.126:41613)
0924 20:24:00.810014 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:00.810019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:00.811415 (+ 1396us) server_negotiation.cc:407] Connection header received
0924 20:24:00.813574 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:00.813580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:00.813648 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:00.813748 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:00.814686 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.815321 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.817243 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.817464 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.817544 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.819093 (+ 1549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:00.819096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:00.819098 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:00.819112 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:00.819131 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:00.819138 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:00.819196 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:00.819416 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 20:24:00.819473 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:24:00.821013 4605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:00.810390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41613 (local address 127.18.172.66:47901)
0924 20:24:00.811249 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:00.811308 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:00.813298 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:00.813924 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:00.813931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:00.814025 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:00.814551 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.814565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.815460 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.815467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.817103 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:00.817120 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:00.817641 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:00.817648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:00.817933 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:00.818660 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:00.818662 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:00.818736 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:00.818845 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:00.819232 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:00.819245 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:00.819253 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:24:00.819414 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 20:24:00.819791 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":147,"threads_started":1}
I20260924 20:24:00.822299 4603 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41613
I20260924 20:24:00.822589 4603 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:00.823359 4603 heartbeater.cc:511] Master 127.18.172.126:41613 requested a full tablet report, sending...
I20260924 20:24:00.824573 4284 ts_manager.cc:194] Registered new tserver with Master: e9a6a30d0377499380ee4a819418b106 (127.18.172.66:38321)
I20260924 20:24:00.825207 4284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.66:47901
W20260924 20:24:00.938268 4606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:00.938586 4606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:00.938673 4606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:00.942943 4606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:00.943042 4606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:00.943135 4606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:24:00.948045 4606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:00.949120 4606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:00.950471 4606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:00.960268 4611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:00.960253 4614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.960494 4606 server_base.cc:1034] running on GCE node
W20260924 20:24:00.960316 4612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:00.961103 4606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:00.961781 4606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:00.963085 4606 hybrid_clock.cc:648] HybridClock initialized: now 1790281440963036 us; error 54 us; skew 500 ppm
I20260924 20:24:00.964897 4606 webserver.cc:460] Webserver started at http://127.18.172.67:36579/ using document root <none> and password file <none>
I20260924 20:24:00.965509 4606 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:00.965571 4606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:00.965811 4606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:00.968047 4606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "bf25e892409c4075854a822290a7add5"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "8396f8f7ddf66c4a349ff60739a28263"
server_key_iv: "27629217fb4074519031aa66332bae9d"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.968604 4606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "bf25e892409c4075854a822290a7add5"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "8396f8f7ddf66c4a349ff60739a28263"
server_key_iv: "27629217fb4074519031aa66332bae9d"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.972422 4606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:24:00.974895 4620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.976125 4606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:00.976264 4606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "bf25e892409c4075854a822290a7add5"
format_stamp: "Formatted at 2026-09-24 20:24:00 on dist-test-slave-m1qq"
server_key: "8396f8f7ddf66c4a349ff60739a28263"
server_key_iv: "27629217fb4074519031aa66332bae9d"
server_key_version: "encryptionkey@0"
I20260924 20:24:00.976392 4606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:00.989691 4606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:00.990407 4606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:00.990597 4606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:00.991323 4606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:00.992331 4606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:00.992398 4606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:00.992473 4606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:00.992514 4606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:01.001224 4606 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:42941
I20260924 20:24:01.001259 4733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:42941 every 8 connection(s)
I20260924 20:24:01.002103 4606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:24:01.009267 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4606
I20260924 20:24:01.009409 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:24:01.009738 19121 external_mini_cluster.cc:1428] Setting key a9bcd2ddf7dc46601eb5dc2d1388a849
I20260924 20:24:01.011169 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.003898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:51665 (local address 127.18.172.126:41613)
0924 20:24:01.004027 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.004030 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.005089 (+ 1059us) server_negotiation.cc:407] Connection header received
0924 20:24:01.006377 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.006380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.006428 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.006503 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:01.007382 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.007927 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.009021 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.009233 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.009336 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.010610 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:01.010614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:01.010617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:01.010629 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:01.010653 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:01.010663 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:01.010749 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.010922 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.010980 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:24:01.012149 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.004317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:41613 (local address 127.18.172.67:51665)
0924 20:24:01.004928 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.004965 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.006195 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.006653 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.006659 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.006739 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:01.007258 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.007271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.008036 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.008038 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.008915 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.008923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.009401 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.009405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.009607 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.010289 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:01.010290 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:01.010379 (+ 89us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:01.010458 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:01.010763 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:01.010778 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:01.010783 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.010888 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.011215 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":107,"threads_started":1}
I20260924 20:24:01.013347 4734 heartbeater.cc:347] Connected to a master server at 127.18.172.126:41613
I20260924 20:24:01.013576 4734 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:01.014091 4734 heartbeater.cc:511] Master 127.18.172.126:41613 requested a full tablet report, sending...
I20260924 20:24:01.015120 4284 ts_manager.cc:194] Registered new tserver with Master: bf25e892409c4075854a822290a7add5 (127.18.172.67:42941)
I20260924 20:24:01.015669 4284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.67:51665
I20260924 20:24:01.023869 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:24:01.029501 4323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.025904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.18.172.126:41613)
0924 20:24:01.026025 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.026034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.026243 (+ 209us) server_negotiation.cc:407] Connection header received
0924 20:24:01.026326 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.026328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.026372 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.026443 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:01.026691 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.027300 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.028091 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.028296 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.028370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.029169 (+ 799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:01.029172 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:01.029174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:01.029184 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:01.029198 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:01.029204 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:01.029256 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.029349 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.029395 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 20:24:01.032112 4284 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:24:01.035274 4284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:24:01.050288 4746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.044972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:42941 (local address 127.0.0.1:55162)
0924 20:24:01.045925 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.045938 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.046042 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.046486 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.046489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.046507 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:01.046586 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.046593 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047681 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.047687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.048850 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.048860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.049724 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.049728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.049841 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.049882 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.050001 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.050121 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":53,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260924 20:24:01.050307 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.045470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.18.172.67:42941)
0924 20:24:01.045849 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.045861 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.046046 (+ 185us) server_negotiation.cc:407] Connection header received
0924 20:24:01.046145 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.046150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.046337 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.046500 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:01.046680 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047546 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.048998 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.049604 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.049713 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.049906 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.050043 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.050144 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":188,"threads_started":1}
I20260924 20:24:01.051287 4749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.044101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:38321 (local address 127.0.0.1:59716)
0924 20:24:01.045410 (+ 1309us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.045425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.045526 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.046138 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.046142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.046160 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:01.046255 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.046260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047733 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.047736 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.049417 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.049424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.050780 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.050784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.050924 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.050951 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.051094 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.051163 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1244,"mutex_wait_us":302,"thread_start_us":120,"threads_started":1}
I20260924 20:24:01.051824 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.044316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.18.172.66:38321)
0924 20:24:01.044640 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.044654 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.045523 (+ 869us) server_negotiation.cc:407] Connection header received
0924 20:24:01.045632 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.045637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.045815 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.045962 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:01.046375 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047598 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.049553 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.050440 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.050551 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.050714 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.051551 (+ 837us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.051657 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":132,"threads_started":1}
I20260924 20:24:01.051930 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.044078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:35797 (local address 127.0.0.1:55124)
0924 20:24:01.044861 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.044911 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.045036 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.045954 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.045964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.046000 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:01.046158 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.046164 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047689 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.047693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.049412 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.049424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.050703 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.050706 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.051633 (+ 927us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.051662 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.051745 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.051812 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":235,"thread_start_us":74,"threads_started":1}
I20260924 20:24:01.052979 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.044887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55124 (local address 127.18.172.65:35797)
0924 20:24:01.045227 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.045237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.045281 (+ 44us) server_negotiation.cc:407] Connection header received
0924 20:24:01.045359 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.045365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.045601 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.045762 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:01.046286 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.047542 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.049558 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.050346 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.050454 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.050640 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.051795 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.052826 (+ 1031us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":124,"threads_started":1}
I20260924 20:24:01.053700 4668 tablet_service.cc:1467] Processing CreateTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe (DEFAULT_TABLE table=test-table [id=c33af6da84b74df69b08777f302f6c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:01.054365 4537 tablet_service.cc:1467] Processing CreateTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe (DEFAULT_TABLE table=test-table [id=c33af6da84b74df69b08777f302f6c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:01.054869 4668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed04fbdf32b543ca8b2a8266978e73fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:01.055322 4537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed04fbdf32b543ca8b2a8266978e73fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:01.055693 4406 tablet_service.cc:1467] Processing CreateTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe (DEFAULT_TABLE table=test-table [id=c33af6da84b74df69b08777f302f6c21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:01.056638 4406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed04fbdf32b543ca8b2a8266978e73fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:01.061544 4751 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Bootstrap starting.
I20260924 20:24:01.063315 4752 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Bootstrap starting.
I20260924 20:24:01.063442 4753 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Bootstrap starting.
I20260924 20:24:01.064224 4751 tablet_bootstrap.cc:654] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:01.065097 4751 log.cc:824] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:01.065158 4752 tablet_bootstrap.cc:654] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:01.065562 4753 tablet_bootstrap.cc:654] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:01.066210 4752 log.cc:824] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:01.066365 4753 log.cc:824] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:01.067595 4751 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: No bootstrap required, opened a new log
I20260924 20:24:01.067790 4751 ts_tablet_manager.cc:1397] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260924 20:24:01.068356 4752 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: No bootstrap required, opened a new log
I20260924 20:24:01.068542 4752 ts_tablet_manager.cc:1397] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 20:24:01.072038 4753 tablet_bootstrap.cc:492] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: No bootstrap required, opened a new log
I20260924 20:24:01.072263 4753 ts_tablet_manager.cc:1397] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260924 20:24:01.072353 4751 raft_consensus.cc:348] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.072710 4751 raft_consensus.cc:374] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:01.072759 4751 raft_consensus.cc:729] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a6a30d0377499380ee4a819418b106, State: Initialized, Role: FOLLOWER
I20260924 20:24:01.072609 4752 raft_consensus.cc:348] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.072840 4752 raft_consensus.cc:374] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:01.072888 4752 raft_consensus.cc:729] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf25e892409c4075854a822290a7add5, State: Initialized, Role: FOLLOWER
I20260924 20:24:01.073275 4751 consensus_queue.cc:260] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.073316 4752 consensus_queue.cc:260] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.074056 4603 heartbeater.cc:503] Master 127.18.172.126:41613 was elected leader, sending a full tablet report...
I20260924 20:24:01.074311 4752 ts_tablet_manager.cc:1428] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:24:01.074339 4734 heartbeater.cc:503] Master 127.18.172.126:41613 was elected leader, sending a full tablet report...
I20260924 20:24:01.075388 4751 ts_tablet_manager.cc:1428] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260924 20:24:01.076539 4753 raft_consensus.cc:348] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.076828 4753 raft_consensus.cc:374] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:01.076906 4753 raft_consensus.cc:729] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 626e8a76920b4a0a8b33ce67936000c2, State: Initialized, Role: FOLLOWER
I20260924 20:24:01.077373 4753 consensus_queue.cc:260] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.078088 4472 heartbeater.cc:503] Master 127.18.172.126:41613 was elected leader, sending a full tablet report...
I20260924 20:24:01.078414 4753 ts_tablet_manager.cc:1428] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
W20260924 20:24:01.121160 4473 tablet.cc:2369] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:01.231215 4758 raft_consensus.cc:482] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:01.231438 4758 raft_consensus.cc:504] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.232676 4758 leader_election.cc:288] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 626e8a76920b4a0a8b33ce67936000c2 (127.18.172.65:35797), e9a6a30d0377499380ee4a819418b106 (127.18.172.66:38321)
I20260924 20:24:01.236825 4748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.232986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:53397 (local address 127.18.172.65:35797)
0924 20:24:01.233142 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.233147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.233164 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:24:01.233221 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.233224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.233279 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.233353 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:01.233684 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.234510 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.235819 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.236231 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.236362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.236413 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.236600 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.236686 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 20:24:01.236825 4736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.232846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:35797 (local address 127.18.172.67:53397)
0924 20:24:01.232990 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.233008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.233101 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.233393 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.233397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.233423 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:01.233562 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.233569 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.234668 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.234672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.235673 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.235683 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.236358 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.236362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.236461 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.236490 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.236581 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.236659 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":57}
I20260924 20:24:01.236835 4747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.233153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:46395 (local address 127.18.172.66:38321)
0924 20:24:01.233263 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.233267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.233454 (+ 187us) server_negotiation.cc:407] Connection header received
0924 20:24:01.233579 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.233581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.233643 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.233838 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:01.234066 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.234748 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.235843 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.236260 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.236398 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.236445 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.236637 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.236709 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:24:01.236851 4760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.232983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:38321 (local address 127.18.172.67:46395)
0924 20:24:01.233379 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.233392 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:01.233479 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:01.233847 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:01.233849 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:01.233866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:01.233958 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.233963 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.234858 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.234860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.235713 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:01.235723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.236410 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.236413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:01.236516 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.236533 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:24:01.236620 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:24:01.236688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"spinlock_wait_cycles":1792,"thread_start_us":123,"threads_started":1}
I20260924 20:24:01.237648 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed04fbdf32b543ca8b2a8266978e73fe" candidate_uuid: "bf25e892409c4075854a822290a7add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a6a30d0377499380ee4a819418b106" is_pre_election: true
I20260924 20:24:01.237639 4426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed04fbdf32b543ca8b2a8266978e73fe" candidate_uuid: "bf25e892409c4075854a822290a7add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e8a76920b4a0a8b33ce67936000c2" is_pre_election: true
I20260924 20:24:01.238013 4557 raft_consensus.cc:2454] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf25e892409c4075854a822290a7add5 in term 0.
I20260924 20:24:01.238013 4426 raft_consensus.cc:2454] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf25e892409c4075854a822290a7add5 in term 0.
I20260924 20:24:01.238622 4622 leader_election.cc:302] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 626e8a76920b4a0a8b33ce67936000c2, bf25e892409c4075854a822290a7add5, e9a6a30d0377499380ee4a819418b106; no voters:
I20260924 20:24:01.238973 4758 raft_consensus.cc:2781] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:01.239075 4758 raft_consensus.cc:482] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:01.239142 4758 raft_consensus.cc:3037] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:01.240180 4758 raft_consensus.cc:504] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.240567 4758 leader_election.cc:288] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [CANDIDATE]: Term 1 election: Requested vote from peers 626e8a76920b4a0a8b33ce67936000c2 (127.18.172.65:35797), e9a6a30d0377499380ee4a819418b106 (127.18.172.66:38321)
I20260924 20:24:01.241123 4426 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed04fbdf32b543ca8b2a8266978e73fe" candidate_uuid: "bf25e892409c4075854a822290a7add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "626e8a76920b4a0a8b33ce67936000c2"
I20260924 20:24:01.241123 4557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed04fbdf32b543ca8b2a8266978e73fe" candidate_uuid: "bf25e892409c4075854a822290a7add5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9a6a30d0377499380ee4a819418b106"
I20260924 20:24:01.241263 4557 raft_consensus.cc:3037] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:01.241263 4426 raft_consensus.cc:3037] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:01.242596 4426 raft_consensus.cc:2454] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf25e892409c4075854a822290a7add5 in term 1.
I20260924 20:24:01.242596 4557 raft_consensus.cc:2454] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf25e892409c4075854a822290a7add5 in term 1.
I20260924 20:24:01.242938 4622 leader_election.cc:302] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 626e8a76920b4a0a8b33ce67936000c2, bf25e892409c4075854a822290a7add5, e9a6a30d0377499380ee4a819418b106; no voters:
I20260924 20:24:01.243115 4758 raft_consensus.cc:2781] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:01.243331 4758 raft_consensus.cc:686] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 1 LEADER]: Becoming Leader. State: Replica: bf25e892409c4075854a822290a7add5, State: Running, Role: LEADER
I20260924 20:24:01.243682 4758 consensus_queue.cc:237] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } }
I20260924 20:24:01.247143 4283 catalog_manager.cc:5495] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf25e892409c4075854a822290a7add5 (127.18.172.67). New cstate: current_term: 1 leader_uuid: "bf25e892409c4075854a822290a7add5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf25e892409c4075854a822290a7add5" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 42941 } health_report { overall_health: HEALTHY } } }
W20260924 20:24:01.253048 4735 tablet.cc:2369] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:01.286057 4750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.282682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.18.172.67:42941)
0924 20:24:01.282818 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:01.282821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:01.282835 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:24:01.282925 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:01.282928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:01.282983 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:01.283121 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:01.283339 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.283841 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.284705 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:01.284916 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:01.284989 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:01.285243 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:01.285677 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:01.285766 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:01.285822 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 20:24:01.285871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:24:01.292235 4557 raft_consensus.cc:1261] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 1 FOLLOWER]: Refusing update from remote peer bf25e892409c4075854a822290a7add5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:01.292235 4426 raft_consensus.cc:1261] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2 [term 1 FOLLOWER]: Refusing update from remote peer bf25e892409c4075854a822290a7add5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:01.293031 4758 consensus_queue.cc:1035] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9a6a30d0377499380ee4a819418b106" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:01.293229 4761 consensus_queue.cc:1035] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "626e8a76920b4a0a8b33ce67936000c2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 35797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:01.301682 4766 mvcc.cc:205] Tried to move back new op lower bound from 7332992783524904960 to 7332992783336898560. Current Snapshot: MvccSnapshot[applied={T|T < 7332992783524904960}]
I20260924 20:24:01.309535 4283 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:01.309693 4283 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:01.312107 4283 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c67d2ad5f1b542e8a0921ae96ad13741: Sending DeleteTablet for 3 replicas of tablet ed04fbdf32b543ca8b2a8266978e73fe
I20260924 20:24:01.312898 4406 tablet_service.cc:1514] Processing DeleteTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:01 UTC) from {username='slave'} at 127.0.0.1:55124
I20260924 20:24:01.313036 4668 tablet_service.cc:1514] Processing DeleteTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:01 UTC) from {username='slave'} at 127.0.0.1:55162
I20260924 20:24:01.313083 4537 tablet_service.cc:1514] Processing DeleteTablet for tablet ed04fbdf32b543ca8b2a8266978e73fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:01 UTC) from {username='slave'} at 127.0.0.1:59716
I20260924 20:24:01.313576 4774 tablet_replica.cc:330] T ed04fbdf32b543ca8b2a8266978e73fe P 626e8a76920b4a0a8b33ce67936000c2: stopping tablet replica
I20260924 20:24:01.313596 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4344
I20260924 20:24:01.314308 4776 tablet_replica.cc:330] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: stopping tablet replica
I20260924 20:24:01.314586 4776 raft_consensus.cc:2229] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:01.314898 4776 raft_consensus.cc:2258] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:01.316803 4776 ts_tablet_manager.cc:1905] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:01.319206 4775 tablet_replica.cc:330] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: stopping tablet replica
I20260924 20:24:01.319350 4776 ts_tablet_manager.cc:1918] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:01.319506 4776 log.cc:1196] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/ed04fbdf32b543ca8b2a8266978e73fe
I20260924 20:24:01.319537 4775 raft_consensus.cc:2229] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:24:01.319952 4776 ts_tablet_manager.cc:1939] T ed04fbdf32b543ca8b2a8266978e73fe P e9a6a30d0377499380ee4a819418b106: Deleting consensus metadata
I20260924 20:24:01.319993 4775 raft_consensus.cc:2258] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:01.321545 4269 catalog_manager.cc:4841] TS e9a6a30d0377499380ee4a819418b106 (127.18.172.66:38321): tablet ed04fbdf32b543ca8b2a8266978e73fe (table test-table [id=c33af6da84b74df69b08777f302f6c21]) successfully deleted
I20260924 20:24:01.321692 4775 ts_tablet_manager.cc:1905] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:24:01.322053 4270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:35797 (error 108)
I20260924 20:24:01.323145 4745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:01.322780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:35797 (local address 127.0.0.1:55136)
0924 20:24:01.322938 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:01.323050 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:35797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 20:24:01.323167 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4475
I20260924 20:24:01.324498 4775 ts_tablet_manager.cc:1918] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:01.324610 4775 log.cc:1196] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/ed04fbdf32b543ca8b2a8266978e73fe
I20260924 20:24:01.324950 4775 ts_tablet_manager.cc:1939] T ed04fbdf32b543ca8b2a8266978e73fe P bf25e892409c4075854a822290a7add5: Deleting consensus metadata
I20260924 20:24:01.326062 4268 catalog_manager.cc:4841] TS bf25e892409c4075854a822290a7add5 (127.18.172.67:42941): tablet ed04fbdf32b543ca8b2a8266978e73fe (table test-table [id=c33af6da84b74df69b08777f302f6c21]) successfully deleted
W20260924 20:24:01.326746 4270 catalog_manager.cc:4584] TS 626e8a76920b4a0a8b33ce67936000c2 (127.18.172.65:35797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed04fbdf32b543ca8b2a8266978e73fe: Network error: Client connection negotiation failed: client connection to 127.18.172.65:35797: connect: Connection refused (error 111)
I20260924 20:24:01.332484 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4606
I20260924 20:24:01.338637 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4253
2026-09-24T20:24:01Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1210 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:01 dist-test-slave-m1qq krb5kdc[4780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:01 dist-test-slave-m1qq krb5kdc[4780](info): set up 2 sockets
Sep 24 20:24:01 dist-test-slave-m1qq krb5kdc[4780](info): commencing operation
krb5kdc: starting...
W20260924 20:24:02.392747 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:02 dist-test-slave-m1qq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:02.518414 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43827
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:02.628875 4793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:02.629190 4793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:02.629271 4793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:02.633407 4793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:02.633504 4793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:02.633545 4793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:02.633606 4793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:02.633647 4793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:02.638967 4793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:02.640151 4793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:02.641196 4793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:02.649682 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:02.649715 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:02.649797 4793 server_base.cc:1034] running on GCE node
W20260924 20:24:02.649972 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:02.650557 4793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:02.651799 4793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:02.652967 4793 hybrid_clock.cc:648] HybridClock initialized: now 1790281442652953 us; error 25 us; skew 500 ppm
Sep 24 20:24:02 dist-test-slave-m1qq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:02.655666 4793 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:02.656543 4793 webserver.cc:460] Webserver started at http://127.18.172.126:46061/ using document root <none> and password file <none>
I20260924 20:24:02.657052 4793 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:02.657111 4793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:02.657274 4793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:02.659350 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "9f90900a989249dd9a751f361f90505f"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "e549b0a95db84aed799bdcef51340d13"
server_key_iv: "179282147ffe48653d4ca09cacd5b655"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.660007 4793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "9f90900a989249dd9a751f361f90505f"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "e549b0a95db84aed799bdcef51340d13"
server_key_iv: "179282147ffe48653d4ca09cacd5b655"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.663612 4793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:02.665934 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:02.667240 4793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:02.667375 4793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "9f90900a989249dd9a751f361f90505f"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "e549b0a95db84aed799bdcef51340d13"
server_key_iv: "179282147ffe48653d4ca09cacd5b655"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.667488 4793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:02.678846 4793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:02.681896 4793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:02.682119 4793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:02.689337 4793 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43827
I20260924 20:24:02.689363 4860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43827 every 8 connection(s)
I20260924 20:24:02.690274 4793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:02.693068 4861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:02.694590 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4793
I20260924 20:24:02.694733 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:02.695116 19121 external_mini_cluster.cc:1428] Setting key cf639a83779260c753b1f6c57b1e2739
I20260924 20:24:02.699416 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: Bootstrap starting.
I20260924 20:24:02.701589 4861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:02.702315 4861 log.cc:824] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:24:02 dist-test-slave-m1qq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:02.704421 4861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: No bootstrap required, opened a new log
I20260924 20:24:02.709297 4861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } }
I20260924 20:24:02.709573 4861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:02.709657 4861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f90900a989249dd9a751f361f90505f, State: Initialized, Role: FOLLOWER
I20260924 20:24:02.710186 4861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } }
I20260924 20:24:02.710345 4861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:02.710397 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:02.696594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51334 (local address 127.18.172.126:43827)
0924 20:24:02.697076 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:02.697091 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:02.697128 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:24:02.697836 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:02.697855 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:02.698195 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:02.698537 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:02.699425 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.700446 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.701337 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.701877 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.702069 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:02.704869 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:02.704875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:02.704892 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:02.704921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:02.707141 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:02.707606 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:02.707612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:02.707618 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:02.707758 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:02.708039 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:02.708044 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:02.708045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:02.708585 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:02.708781 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:02.709004 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:24:02.709338 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":133,"threads_started":1}
I20260924 20:24:02.710415 4861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:02.710533 4861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:02.711596 4861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } }
I20260924 20:24:02.711958 4861 leader_election.cc:302] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f90900a989249dd9a751f361f90505f; no voters:
I20260924 20:24:02.712273 4861 leader_election.cc:288] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:02.712400 4866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:02.712598 4866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [term 1 LEADER]: Becoming Leader. State: Replica: 9f90900a989249dd9a751f361f90505f, State: Running, Role: LEADER
I20260924 20:24:02.712982 4866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } }
I20260924 20:24:02.713516 4861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:02.714632 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f90900a989249dd9a751f361f90505f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } } }
I20260924 20:24:02.714777 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:02.715209 4875 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:02.715170 4868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f90900a989249dd9a751f361f90505f. Latest consensus state: current_term: 1 leader_uuid: "9f90900a989249dd9a751f361f90505f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f90900a989249dd9a751f361f90505f" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43827 } } }
I20260924 20:24:02.715828 4868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:02.718312 4875 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:02.724004 4875 catalog_manager.cc:1347] Generated new cluster ID: 6f2a01b73bfa40babca06cc700f8261c
I20260924 20:24:02.724081 4875 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:02.737771 4875 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:02.739413 4875 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:02.749694 4875 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: Generated new TSK 0
I20260924 20:24:02.750567 4875 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:24:02.780378 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:02.897514 4887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:02.897835 4887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:02.897924 4887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:02.902038 4887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:02.902141 4887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:02.902235 4887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:24:02.907583 4887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:02.908665 4887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:02.909711 4887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:02.919466 4893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:02.919508 4892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:02.919466 4895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:02.919982 4887 server_base.cc:1034] running on GCE node
I20260924 20:24:02.920392 4887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:02.921026 4887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:02.922230 4887 hybrid_clock.cc:648] HybridClock initialized: now 1790281442922186 us; error 50 us; skew 500 ppm
Sep 24 20:24:02 dist-test-slave-m1qq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:02.925235 4887 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:24:02.926141 4887 webserver.cc:460] Webserver started at http://127.18.172.65:39753/ using document root <none> and password file <none>
I20260924 20:24:02.926818 4887 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:02.926928 4887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:02.927237 4887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:02.929477 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "044c630699554e80ac715cd9c4c94095"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "5317de9d32fa13f3e8c55c7eaf8ca6f6"
server_key_iv: "6de363f738b1ff1dcded901f42b18097"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.930094 4887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "044c630699554e80ac715cd9c4c94095"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "5317de9d32fa13f3e8c55c7eaf8ca6f6"
server_key_iv: "6de363f738b1ff1dcded901f42b18097"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.934199 4887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:24:02.936785 4902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:02.938048 4887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:02.938205 4887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "044c630699554e80ac715cd9c4c94095"
format_stamp: "Formatted at 2026-09-24 20:24:02 on dist-test-slave-m1qq"
server_key: "5317de9d32fa13f3e8c55c7eaf8ca6f6"
server_key_iv: "6de363f738b1ff1dcded901f42b18097"
server_key_version: "encryptionkey@0"
I20260924 20:24:02.938324 4887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:02.955739 4887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:02.958870 4887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:02.959129 4887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:02.959745 4887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:02.960705 4887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:02.960777 4887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:02.960845 4887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:02.960893 4887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:02.969327 4887 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:34887
I20260924 20:24:02.969348 5015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:34887 every 8 connection(s)
I20260924 20:24:02.970222 4887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:24:02.977453 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 4887
I20260924 20:24:02.977602 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:24:02.977974 19121 external_mini_cluster.cc:1428] Setting key 793df4b718d039d9c2ef765485a68cdc
Sep 24 20:24:02 dist-test-slave-m1qq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281442, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:02.984985 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:02.972163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:54679 (local address 127.18.172.126:43827)
0924 20:24:02.972308 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:02.972312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:02.973186 (+ 874us) server_negotiation.cc:407] Connection header received
0924 20:24:02.974603 (+ 1417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:02.974607 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:02.974666 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:02.974739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:02.975960 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.976510 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.978012 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.978289 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.978416 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:02.981262 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:02.981271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:02.981273 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:02.981298 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:02.982801 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:02.983410 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:02.983415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:02.983416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:02.983495 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:02.983843 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:02.983847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:02.983848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:02.984035 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:02.984141 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:02.984667 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 20:24:02.984797 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:24:02.985785 5018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:02.972588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43827 (local address 127.18.172.65:54679)
0924 20:24:02.973036 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:02.973080 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:02.974374 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:02.974910 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:02.974917 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:02.975331 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:02.975802 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:02.975815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.976638 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.976648 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:02.977868 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:02.977880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:02.978466 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:02.978469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:02.978673 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:02.979384 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:02.979407 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:02.981091 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:02.982938 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:02.982945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:02.982962 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:02.983289 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:02.983593 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:02.983596 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:02.983598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:02.983740 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:02.984140 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:02.984147 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:02.984417 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 20:24:02.984613 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 20:24:02.984948 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:24:02.987093 5016 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43827
I20260924 20:24:02.987353 5016 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:02.987857 5016 heartbeater.cc:511] Master 127.18.172.126:43827 requested a full tablet report, sending...
I20260924 20:24:02.989359 4825 ts_manager.cc:194] Registered new tserver with Master: 044c630699554e80ac715cd9c4c94095 (127.18.172.65:34887)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
I20260924 20:24:02.990698 4825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:54679
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:24:03.006384 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:03.117156 5021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:03.117404 5021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:03.117449 5021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:03.121500 5021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:03.121567 5021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:03.121626 5021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:24:03.126922 5021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:03.127934 5021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:03.128867 5021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:03.138492 5026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.138576 5027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.138551 5029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:03.138797 5021 server_base.cc:1034] running on GCE node
I20260924 20:24:03.139364 5021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:03.140013 5021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:03.141228 5021 hybrid_clock.cc:648] HybridClock initialized: now 1790281443141196 us; error 52 us; skew 500 ppm
Sep 24 20:24:03 dist-test-slave-m1qq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:03.144496 5021 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:24:03.145529 5021 webserver.cc:460] Webserver started at http://127.18.172.66:36133/ using document root <none> and password file <none>
I20260924 20:24:03.146137 5021 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:03.146239 5021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:03.146481 5021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:03.148686 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "dbb6648b1f7f43fc91c384abc990c4de"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "a9b80c8a463bf71b189f3659e456303d"
server_key_iv: "eb6e85b96819a600cd17beae833fa296"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.149300 5021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "dbb6648b1f7f43fc91c384abc990c4de"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "a9b80c8a463bf71b189f3659e456303d"
server_key_iv: "eb6e85b96819a600cd17beae833fa296"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.153266 5021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:24:03.155797 5036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.157087 5021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:03.157240 5021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "dbb6648b1f7f43fc91c384abc990c4de"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "a9b80c8a463bf71b189f3659e456303d"
server_key_iv: "eb6e85b96819a600cd17beae833fa296"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.157363 5021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:03.171728 5021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:03.174984 5021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:03.175282 5021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:03.175876 5021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:03.176678 5021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:03.176723 5021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.176761 5021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:03.176780 5021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.185669 5021 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:39807
I20260924 20:24:03.185704 5149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:39807 every 8 connection(s)
I20260924 20:24:03.186609 5021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:24:03.193069 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5021
I20260924 20:24:03.193243 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:24:03.193575 19121 external_mini_cluster.cc:1428] Setting key 839226a06c11dd3132b51c73ce7c1a17
Sep 24 20:24:03 dist-test-slave-m1qq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:03.201751 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.188502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:38265 (local address 127.18.172.126:43827)
0924 20:24:03.188666 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.188670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.189584 (+ 914us) server_negotiation.cc:407] Connection header received
0924 20:24:03.191055 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.191059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.191115 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.191196 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:03.192363 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.192946 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.194328 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.194615 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.194724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.197651 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:03.197657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:03.197665 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:03.197690 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:03.199283 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.199844 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.199849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.199851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.199937 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.200325 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.200328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.200330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.200541 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:03.200782 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.201425 (+ 643us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.201571 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 20:24:03.202713 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.188971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43827 (local address 127.18.172.66:38265)
0924 20:24:03.189435 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.189479 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.190760 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.191365 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.191372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.191728 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:03.192217 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.192230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.193076 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.193080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.194179 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.194189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.194765 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.194768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.195041 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.195681 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:03.195697 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:03.197461 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:03.199420 (+ 1959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:03.199426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:03.199444 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:03.199732 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:03.200038 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:03.200041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:03.200043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:03.200218 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:03.200837 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:03.200851 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:03.201147 (+ 296us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.201382 (+ 235us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.201770 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:24:03.203994 5150 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43827
I20260924 20:24:03.204280 5150 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:03.204828 5150 heartbeater.cc:511] Master 127.18.172.126:43827 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:24:03.206067 4825 ts_manager.cc:194] Registered new tserver with Master: dbb6648b1f7f43fc91c384abc990c4de (127.18.172.66:39807)
I20260924 20:24:03.206955 4825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:38265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:24:03.222891 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:03.335585 5155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:03.335867 5155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:03.335939 5155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:03.340185 5155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:03.340286 5155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:03.340384 5155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:24:03.346089 5155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:43827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:03.347332 5155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:03.348452 5155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:24:03.358724 5155 server_base.cc:1034] running on GCE node
W20260924 20:24:03.358637 5163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.358639 5161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.358650 5160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:03.359333 5155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:03.359987 5155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:03.361145 5155 hybrid_clock.cc:648] HybridClock initialized: now 1790281443361105 us; error 51 us; skew 500 ppm
Sep 24 20:24:03 dist-test-slave-m1qq krb5kdc[4780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:03.364002 5155 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:24:03.364948 5155 webserver.cc:460] Webserver started at http://127.18.172.67:44935/ using document root <none> and password file <none>
I20260924 20:24:03.365511 5155 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:03.365573 5155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:03.365815 5155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:03.367924 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "a16cd5c609044a5d88b10a49dd9da8ff"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "afe8b32623e377c905d9b4273ced311a"
server_key_iv: "c72f4a27c994d1b309af1f1a2cb4a460"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.368489 5155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "a16cd5c609044a5d88b10a49dd9da8ff"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "afe8b32623e377c905d9b4273ced311a"
server_key_iv: "c72f4a27c994d1b309af1f1a2cb4a460"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.372387 5155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:24:03.374953 5170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.376365 5155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:03.376515 5155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "a16cd5c609044a5d88b10a49dd9da8ff"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "afe8b32623e377c905d9b4273ced311a"
server_key_iv: "c72f4a27c994d1b309af1f1a2cb4a460"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.376638 5155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:03.406630 5155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:03.409911 5155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:03.410123 5155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:03.410738 5155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:03.411651 5155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:03.411724 5155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.411788 5155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:03.411839 5155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.421237 5155 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:40625
I20260924 20:24:03.421262 5283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:40625 every 8 connection(s)
I20260924 20:24:03.422122 5155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:24:03.429432 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5155
I20260924 20:24:03.429584 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:24:03.429893 19121 external_mini_cluster.cc:1428] Setting key 85c2990c09c95de32ff39e0d16c71b30
Sep 24 20:24:03 dist-test-slave-m1qq krb5kdc[4780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281443, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:03.435670 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.423993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:55051 (local address 127.18.172.126:43827)
0924 20:24:03.424137 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.424141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.424912 (+ 771us) server_negotiation.cc:407] Connection header received
0924 20:24:03.426282 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.426285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.426346 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.426479 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:03.427724 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.428255 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.429394 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.429721 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.429832 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.432539 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:03.432542 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:03.432544 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:03.432567 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:03.433786 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.434244 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.434246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.434248 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.434316 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.434641 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.434643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.434645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.434863 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:03.434941 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.435462 (+ 521us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.435554 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:24:03.436570 5286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.424308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43827 (local address 127.18.172.67:55051)
0924 20:24:03.424750 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.424796 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.426065 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.426637 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.426644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.427047 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:03.427571 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.427585 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.428386 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.428394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.429267 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.429276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.429874 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.429878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.430095 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.430706 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:03.430722 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:03.432388 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:03.433889 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:03.433893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:03.433905 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:03.434159 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:03.434416 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:03.434418 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:03.434420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:03.434553 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:03.434952 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:03.434957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:03.435235 (+ 278us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.435401 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.435665 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260924 20:24:03.437736 5284 heartbeater.cc:347] Connected to a master server at 127.18.172.126:43827
I20260924 20:24:03.438015 5284 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:03.438496 5284 heartbeater.cc:511] Master 127.18.172.126:43827 requested a full tablet report, sending...
I20260924 20:24:03.439586 4825 ts_manager.cc:194] Registered new tserver with Master: a16cd5c609044a5d88b10a49dd9da8ff (127.18.172.67:40625)
I20260924 20:24:03.440129 4825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:55051
I20260924 20:24:03.443895 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:24:03.454023 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.445850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51350 (local address 127.18.172.126:43827)
0924 20:24:03.446004 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.446014 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.446138 (+ 124us) server_negotiation.cc:407] Connection header received
0924 20:24:03.446288 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.446291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.446333 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.446420 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:03.447211 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.448001 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.448918 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.449154 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.449280 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.450910 (+ 1630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:03.450919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:03.450921 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:03.450943 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:03.452196 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.452700 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.452704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.452706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.452787 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:03.453115 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:03.453118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:03.453120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:03.453326 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:03.453488 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.453759 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.453857 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:24:03.457336 4825 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:24:03.460394 4825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:24:03.475631 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.469331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49222 (local address 127.18.172.67:40625)
0924 20:24:03.470894 (+ 1563us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.470929 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.470953 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:24:03.471051 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.471057 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.471647 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.471846 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:03.472052 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.472836 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.474280 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.474925 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.475065 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.475247 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.475381 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.475481 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":156,"threads_started":1}
I20260924 20:24:03.475626 5295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.469128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:40625 (local address 127.0.0.1:49222)
0924 20:24:03.469549 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.469603 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.469729 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.471798 (+ 2069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.471803 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.471825 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:03.471952 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.471961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.473012 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.473015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.474150 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.474159 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.475062 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.475066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.475181 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.475225 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.475338 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.475469 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":144,"threads_started":1}
I20260924 20:24:03.476595 5297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.469926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:39807 (local address 127.0.0.1:41268)
0924 20:24:03.471311 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.471329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.471438 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.472006 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.472010 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.472031 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:03.472115 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.472121 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.473456 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.473460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.474739 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.474750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.476007 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.476011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.476096 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.476141 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.476215 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.476417 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1323,"mutex_wait_us":62,"thread_start_us":80,"threads_started":1}
I20260924 20:24:03.476874 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.469955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.18.172.65:34887)
0924 20:24:03.470340 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.470355 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.470645 (+ 290us) server_negotiation.cc:407] Connection header received
0924 20:24:03.470928 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.470936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.471194 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.471364 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:03.471904 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.473167 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.474875 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.475830 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.476011 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.476195 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.476598 (+ 403us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.476720 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":87,"threads_started":1}
I20260924 20:24:03.477026 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.470115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41268 (local address 127.18.172.66:39807)
0924 20:24:03.470611 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.470626 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.471452 (+ 826us) server_negotiation.cc:407] Connection header received
0924 20:24:03.471538 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.471543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.471692 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.471846 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:03.472240 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.473318 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.474884 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.475755 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.475855 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.475992 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.476772 (+ 780us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.476867 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":135,"threads_started":1}
I20260924 20:24:03.477406 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.469552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34887 (local address 127.0.0.1:40792)
0924 20:24:03.470542 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.470556 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.470722 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.471540 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.471545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.471612 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:03.471768 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.471779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.473370 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.473376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.474743 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.474750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.476293 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.476302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.476419 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.476441 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.477186 (+ 745us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.477277 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":76,"thread_start_us":93,"threads_started":1}
I20260924 20:24:03.479346 5218 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 (DEFAULT_TABLE table=test-table [id=1e4a57952b694dc69394239d707ab012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:03.479873 4950 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 (DEFAULT_TABLE table=test-table [id=1e4a57952b694dc69394239d707ab012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:03.479987 5084 tablet_service.cc:1467] Processing CreateTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 (DEFAULT_TABLE table=test-table [id=1e4a57952b694dc69394239d707ab012]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:03.480577 5218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0e8f96f0d4b409316924fbb6243c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:03.480947 4950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0e8f96f0d4b409316924fbb6243c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:03.480978 5084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1e0e8f96f0d4b409316924fbb6243c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:03.487361 5301 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: Bootstrap starting.
I20260924 20:24:03.488641 5302 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Bootstrap starting.
I20260924 20:24:03.489585 5301 tablet_bootstrap.cc:654] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:03.490342 5301 log.cc:824] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:03.490490 5302 tablet_bootstrap.cc:654] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:03.491384 5303 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Bootstrap starting.
I20260924 20:24:03.491786 5302 log.cc:824] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:03.493052 5301 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: No bootstrap required, opened a new log
I20260924 20:24:03.493270 5301 ts_tablet_manager.cc:1397] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:24:03.493670 5303 tablet_bootstrap.cc:654] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:03.493984 5302 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: No bootstrap required, opened a new log
I20260924 20:24:03.494197 5302 ts_tablet_manager.cc:1397] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:24:03.494971 5303 log.cc:824] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:03.497051 5303 tablet_bootstrap.cc:492] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: No bootstrap required, opened a new log
I20260924 20:24:03.497238 5303 ts_tablet_manager.cc:1397] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260924 20:24:03.497080 5301 raft_consensus.cc:348] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.497438 5301 raft_consensus.cc:374] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:03.497521 5301 raft_consensus.cc:729] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 044c630699554e80ac715cd9c4c94095, State: Initialized, Role: FOLLOWER
I20260924 20:24:03.498013 5301 consensus_queue.cc:260] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.498072 5302 raft_consensus.cc:348] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.498385 5302 raft_consensus.cc:374] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:03.498489 5302 raft_consensus.cc:729] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb6648b1f7f43fc91c384abc990c4de, State: Initialized, Role: FOLLOWER
I20260924 20:24:03.498968 5302 consensus_queue.cc:260] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.499214 5301 ts_tablet_manager.cc:1428] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:24:03.499536 5016 heartbeater.cc:503] Master 127.18.172.126:43827 was elected leader, sending a full tablet report...
I20260924 20:24:03.499837 5150 heartbeater.cc:503] Master 127.18.172.126:43827 was elected leader, sending a full tablet report...
I20260924 20:24:03.500010 5302 ts_tablet_manager.cc:1428] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:24:03.502398 5303 raft_consensus.cc:348] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.502646 5303 raft_consensus.cc:374] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:03.502730 5303 raft_consensus.cc:729] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a16cd5c609044a5d88b10a49dd9da8ff, State: Initialized, Role: FOLLOWER
I20260924 20:24:03.503219 5303 consensus_queue.cc:260] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.504310 5303 ts_tablet_manager.cc:1428] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:24:03.504467 5284 heartbeater.cc:503] Master 127.18.172.126:43827 was elected leader, sending a full tablet report...
I20260924 20:24:03.539455 5307 raft_consensus.cc:482] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:03.539738 5307 raft_consensus.cc:504] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.540946 5307 leader_election.cc:288] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a16cd5c609044a5d88b10a49dd9da8ff (127.18.172.67:40625), dbb6648b1f7f43fc91c384abc990c4de (127.18.172.66:39807)
I20260924 20:24:03.545737 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.541457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:33123 (local address 127.18.172.67:40625)
0924 20:24:03.541592 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.541596 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.541945 (+ 349us) server_negotiation.cc:407] Connection header received
0924 20:24:03.542163 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.542168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.542236 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.542332 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:03.542657 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.543311 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.544613 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.545049 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.545194 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.545289 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.545518 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.545593 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:24:03.545778 5300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.542039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:56945 (local address 127.18.172.66:39807)
0924 20:24:03.542167 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.542173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.542277 (+ 104us) server_negotiation.cc:407] Connection header received
0924 20:24:03.542450 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.542453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.542551 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.542624 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:03.542902 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.543516 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.544744 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.545162 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.545339 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.545398 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.545595 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.545666 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:24:03.545801 5310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.541340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:40625 (local address 127.18.172.65:33123)
0924 20:24:03.541834 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.541852 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.541993 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.542356 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.542359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.542383 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:03.542529 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.542535 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.543438 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.543442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.544472 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.544482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.545195 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.545198 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.545302 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.545388 (+ 86us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.545501 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.545583 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":58,"thread_start_us":193,"threads_started":1}
I20260924 20:24:03.545805 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.541852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:39807 (local address 127.18.172.65:56945)
0924 20:24:03.542167 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.542182 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:03.542330 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:03.542638 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:03.542641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:03.542658 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:03.542784 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.542790 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.543627 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.543633 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.544620 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:03.544632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.545319 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.545323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:03.545468 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.545493 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 20:24:03.545579 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:24:03.545648 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260924 20:24:03.546512 5104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0e8f96f0d4b409316924fbb6243c9" candidate_uuid: "044c630699554e80ac715cd9c4c94095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb6648b1f7f43fc91c384abc990c4de" is_pre_election: true
I20260924 20:24:03.546514 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0e8f96f0d4b409316924fbb6243c9" candidate_uuid: "044c630699554e80ac715cd9c4c94095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" is_pre_election: true
I20260924 20:24:03.546849 5104 raft_consensus.cc:2454] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 044c630699554e80ac715cd9c4c94095 in term 0.
I20260924 20:24:03.546869 5238 raft_consensus.cc:2454] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 044c630699554e80ac715cd9c4c94095 in term 0.
I20260924 20:24:03.547386 4903 leader_election.cc:302] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 044c630699554e80ac715cd9c4c94095, a16cd5c609044a5d88b10a49dd9da8ff; no voters:
I20260924 20:24:03.547684 5307 raft_consensus.cc:2781] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:03.547768 5307 raft_consensus.cc:482] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:03.547803 5307 raft_consensus.cc:3037] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:03.548966 5307 raft_consensus.cc:504] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.549353 5307 leader_election.cc:288] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [CANDIDATE]: Term 1 election: Requested vote from peers a16cd5c609044a5d88b10a49dd9da8ff (127.18.172.67:40625), dbb6648b1f7f43fc91c384abc990c4de (127.18.172.66:39807)
I20260924 20:24:03.549962 5238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0e8f96f0d4b409316924fbb6243c9" candidate_uuid: "044c630699554e80ac715cd9c4c94095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a16cd5c609044a5d88b10a49dd9da8ff"
I20260924 20:24:03.550135 5238 raft_consensus.cc:3037] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:03.550153 5104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1e0e8f96f0d4b409316924fbb6243c9" candidate_uuid: "044c630699554e80ac715cd9c4c94095" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dbb6648b1f7f43fc91c384abc990c4de"
I20260924 20:24:03.550295 5104 raft_consensus.cc:3037] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:03.551457 5238 raft_consensus.cc:2454] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 044c630699554e80ac715cd9c4c94095 in term 1.
I20260924 20:24:03.551537 5104 raft_consensus.cc:2454] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 044c630699554e80ac715cd9c4c94095 in term 1.
I20260924 20:24:03.551836 4903 leader_election.cc:302] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 044c630699554e80ac715cd9c4c94095, dbb6648b1f7f43fc91c384abc990c4de; no voters:
I20260924 20:24:03.552012 5307 raft_consensus.cc:2781] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:03.552270 5307 raft_consensus.cc:686] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [term 1 LEADER]: Becoming Leader. State: Replica: 044c630699554e80ac715cd9c4c94095, State: Running, Role: LEADER
I20260924 20:24:03.552614 5307 consensus_queue.cc:237] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } }
I20260924 20:24:03.556059 4825 catalog_manager.cc:5495] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 reported cstate change: term changed from 0 to 1, leader changed from <none> to 044c630699554e80ac715cd9c4c94095 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "044c630699554e80ac715cd9c4c94095" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "044c630699554e80ac715cd9c4c94095" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 34887 } health_report { overall_health: HEALTHY } } }
I20260924 20:24:03.575783 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.571845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40806 (local address 127.18.172.65:34887)
0924 20:24:03.571999 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.572005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.572023 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:24:03.572222 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.572225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.572279 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.572387 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:03.572642 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.573188 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.574134 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.574365 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.574498 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.574829 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:03.575333 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:03.575510 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.575573 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.575614 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:24:03.582201 5238 raft_consensus.cc:1261] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 1 FOLLOWER]: Refusing update from remote peer 044c630699554e80ac715cd9c4c94095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:03.582201 5104 raft_consensus.cc:1261] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 1 FOLLOWER]: Refusing update from remote peer 044c630699554e80ac715cd9c4c94095: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:03.582903 5307 consensus_queue.cc:1035] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a16cd5c609044a5d88b10a49dd9da8ff" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:03.583192 5312 consensus_queue.cc:1035] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbb6648b1f7f43fc91c384abc990c4de" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:03.591035 5316 mvcc.cc:205] Tried to move back new op lower bound from 7332992792904769536 to 7332992792794304512. Current Snapshot: MvccSnapshot[applied={T|T < 7332992792904769536}]
I20260924 20:24:03.599579 4825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:03.599757 4825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51350:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:03.602309 4825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f90900a989249dd9a751f361f90505f: Sending DeleteTablet for 3 replicas of tablet e1e0e8f96f0d4b409316924fbb6243c9
I20260924 20:24:03.603286 4950 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:40792
I20260924 20:24:03.603327 5218 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:49222
I20260924 20:24:03.603847 5324 tablet_replica.cc:330] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: stopping tablet replica
I20260924 20:24:03.603837 5084 tablet_service.cc:1514] Processing DeleteTablet for tablet e1e0e8f96f0d4b409316924fbb6243c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:03 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:41268
I20260924 20:24:03.604130 5325 tablet_replica.cc:330] T e1e0e8f96f0d4b409316924fbb6243c9 P 044c630699554e80ac715cd9c4c94095: stopping tablet replica
I20260924 20:24:03.604224 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4887
I20260924 20:24:03.604271 5326 tablet_replica.cc:330] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: stopping tablet replica
I20260924 20:24:03.605126 5324 raft_consensus.cc:2229] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:03.605430 5326 raft_consensus.cc:2229] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:03.605479 5324 raft_consensus.cc:2258] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:03.605713 5326 raft_consensus.cc:2258] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:03.606823 5324 ts_tablet_manager.cc:1905] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:03.607160 5326 ts_tablet_manager.cc:1905] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:03.609405 5324 ts_tablet_manager.cc:1918] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:03.609515 5324 log.cc:1196] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/e1e0e8f96f0d4b409316924fbb6243c9
I20260924 20:24:03.609815 5324 ts_tablet_manager.cc:1939] T e1e0e8f96f0d4b409316924fbb6243c9 P a16cd5c609044a5d88b10a49dd9da8ff: Deleting consensus metadata
I20260924 20:24:03.610451 5326 ts_tablet_manager.cc:1918] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:03.610549 5326 log.cc:1196] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/e1e0e8f96f0d4b409316924fbb6243c9
I20260924 20:24:03.610880 5326 ts_tablet_manager.cc:1939] T e1e0e8f96f0d4b409316924fbb6243c9 P dbb6648b1f7f43fc91c384abc990c4de: Deleting consensus metadata
I20260924 20:24:03.611100 4809 catalog_manager.cc:4841] TS a16cd5c609044a5d88b10a49dd9da8ff (127.18.172.67:40625): tablet e1e0e8f96f0d4b409316924fbb6243c9 (table test-table [id=1e4a57952b694dc69394239d707ab012]) successfully deleted
I20260924 20:24:03.612098 4809 catalog_manager.cc:4841] TS dbb6648b1f7f43fc91c384abc990c4de (127.18.172.66:39807): tablet e1e0e8f96f0d4b409316924fbb6243c9 (table test-table [id=1e4a57952b694dc69394239d707ab012]) successfully deleted
W20260924 20:24:03.613201 4811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:34887 (error 108)
I20260924 20:24:03.614089 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5021
I20260924 20:24:03.614297 5298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.613970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:34887 (local address 127.0.0.1:40822)
0924 20:24:03.614116 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:03.614198 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 20:24:03.615067 4811 catalog_manager.cc:4584] TS 044c630699554e80ac715cd9c4c94095 (127.18.172.65:34887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1e0e8f96f0d4b409316924fbb6243c9: Network error: Client connection negotiation failed: client connection to 127.18.172.65:34887: connect: Connection refused (error 111)
I20260924 20:24:03.619688 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5155
I20260924 20:24:03.625063 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 4793
2026-09-24T20:24:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2285 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T20:24:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:03Z Disabled control of system clock
I20260924 20:24:03.655568 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35409
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:35409
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:03.765913 5331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:03.766198 5331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:03.766235 5331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:03.770346 5331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:03.770417 5331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:03.770433 5331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:03.770452 5331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:03.770467 5331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:03.775379 5331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:35409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:35409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:03.776563 5331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:03.778119 5331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:03.787220 5336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.787250 5339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:03.787251 5337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:03.787395 5331 server_base.cc:1034] running on GCE node
I20260924 20:24:03.788228 5331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:03.789381 5331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:03.790592 5331 hybrid_clock.cc:648] HybridClock initialized: now 1790281443790563 us; error 48 us; skew 500 ppm
I20260924 20:24:03.792474 5331 webserver.cc:460] Webserver started at http://127.18.172.126:43451/ using document root <none> and password file <none>
I20260924 20:24:03.793064 5331 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:03.793159 5331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:03.793401 5331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:03.795562 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "45c77cf09bb140bc8e307c11485c15e2"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "19c3226de761317f469cdd56f6228de0"
server_key_iv: "f7dedc88440cf255dbe8f98e61944b4c"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.796123 5331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "45c77cf09bb140bc8e307c11485c15e2"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "19c3226de761317f469cdd56f6228de0"
server_key_iv: "f7dedc88440cf255dbe8f98e61944b4c"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.799817 5331 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:24:03.802186 5345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:03.803519 5331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:03.803661 5331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "45c77cf09bb140bc8e307c11485c15e2"
format_stamp: "Formatted at 2026-09-24 20:24:03 on dist-test-slave-m1qq"
server_key: "19c3226de761317f469cdd56f6228de0"
server_key_iv: "f7dedc88440cf255dbe8f98e61944b4c"
server_key_version: "encryptionkey@0"
I20260924 20:24:03.803782 5331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:03.835716 5331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:03.836472 5331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:03.836680 5331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:03.844115 5331 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:35409
I20260924 20:24:03.844137 5397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:35409 every 8 connection(s)
I20260924 20:24:03.845077 5331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:03.848270 5398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:03.851697 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5331
I20260924 20:24:03.851854 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:03.852139 19121 external_mini_cluster.cc:1428] Setting key 33e90847cd4b1b556cb6f77cdc08a7ca
I20260924 20:24:03.854032 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: Bootstrap starting.
I20260924 20:24:03.856364 5398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:03.857108 5398 log.cc:824] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:03.859257 5398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: No bootstrap required, opened a new log
I20260924 20:24:03.861222 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:03.853552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.18.172.126:35409)
0924 20:24:03.854179 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:03.854203 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:03.854240 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:24:03.854866 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:03.854881 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:03.855363 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:03.855645 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:03.855984 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.857014 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.858107 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:03.858565 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:03.858682 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:03.859536 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:03.859541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:03.859553 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:03.859922 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:03.859976 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:03.859996 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:03.860073 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:03.860227 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 20:24:03.860527 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":126,"threads_started":1}
I20260924 20:24:03.863260 5398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } }
I20260924 20:24:03.863554 5398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:03.863613 5398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45c77cf09bb140bc8e307c11485c15e2, State: Initialized, Role: FOLLOWER
I20260924 20:24:03.863996 5398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } }
I20260924 20:24:03.864109 5398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:03.864154 5398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:03.864220 5398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:03.865036 5398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } }
I20260924 20:24:03.865305 5398 leader_election.cc:302] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45c77cf09bb140bc8e307c11485c15e2; no voters:
I20260924 20:24:03.865542 5398 leader_election.cc:288] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:03.865660 5403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:03.865895 5403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [term 1 LEADER]: Becoming Leader. State: Replica: 45c77cf09bb140bc8e307c11485c15e2, State: Running, Role: LEADER
I20260924 20:24:03.866195 5403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } }
I20260924 20:24:03.866719 5398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:03.867852 5405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45c77cf09bb140bc8e307c11485c15e2. Latest consensus state: current_term: 1 leader_uuid: "45c77cf09bb140bc8e307c11485c15e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } } }
I20260924 20:24:03.867990 5405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:03.868268 5404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45c77cf09bb140bc8e307c11485c15e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45c77cf09bb140bc8e307c11485c15e2" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 35409 } } }
I20260924 20:24:03.868381 5404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:03.868695 5412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:03.871733 5412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:03.877843 5412 catalog_manager.cc:1347] Generated new cluster ID: b33f78cf1f364fcd983e384fd5dbd747
I20260924 20:24:03.877966 5412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:03.907781 5412 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:03.908860 5412 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:03.915733 5412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: Generated new TSK 0
I20260924 20:24:03.916428 5412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:03.928231 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--builtin_ntp_servers=127.18.172.84:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:04.039961 5422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:04.040241 5422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:04.040315 5422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:04.044340 5422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:04.044391 5422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:04.044483 5422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:24:04.049284 5422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:04.050377 5422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:04.051836 5422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:04.061822 5428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.061856 5430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.061858 5427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:04.062172 5422 server_base.cc:1034] running on GCE node
I20260924 20:24:04.062584 5422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:04.063246 5422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:04.064406 5422 hybrid_clock.cc:648] HybridClock initialized: now 1790281444064391 us; error 27 us; skew 500 ppm
I20260924 20:24:04.066177 5422 webserver.cc:460] Webserver started at http://127.18.172.65:45461/ using document root <none> and password file <none>
I20260924 20:24:04.066743 5422 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:04.066805 5422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:04.067106 5422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:04.069247 5422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "6eee4b94383b4fafa7e257e65bda7bf6"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "c5f4dc684c8471c8d869035b90007e30"
server_key_iv: "558a4e3a8b527cbe2a1996fca11f164b"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.069840 5422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "6eee4b94383b4fafa7e257e65bda7bf6"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "c5f4dc684c8471c8d869035b90007e30"
server_key_iv: "558a4e3a8b527cbe2a1996fca11f164b"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.073483 5422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:24:04.075946 5436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.077185 5422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:04.077307 5422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "6eee4b94383b4fafa7e257e65bda7bf6"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "c5f4dc684c8471c8d869035b90007e30"
server_key_iv: "558a4e3a8b527cbe2a1996fca11f164b"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.077419 5422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:04.096114 5422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:04.096817 5422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:04.097023 5422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:04.097617 5422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:04.098563 5422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:04.098608 5422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.098680 5422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:04.098711 5422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.107900 5422 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:39777
I20260924 20:24:04.107923 5549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:39777 every 8 connection(s)
I20260924 20:24:04.108803 5422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:24:04.115370 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5422
I20260924 20:24:04.115481 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:24:04.115759 19121 external_mini_cluster.cc:1428] Setting key efdef64266ae5be2f2432971ba2a541a
I20260924 20:24:04.117619 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--builtin_ntp_servers=127.18.172.84:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:04.118081 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.110583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:55491 (local address 127.18.172.126:35409)
0924 20:24:04.110759 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.110762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.111894 (+ 1132us) server_negotiation.cc:407] Connection header received
0924 20:24:04.113333 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.113336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.113383 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.113467 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:04.114331 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.114849 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.116130 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.116370 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.116488 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.117562 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:04.117566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:04.117569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:04.117582 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:04.117602 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:04.117617 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:04.117694 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.117920 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.117971 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:24:04.119288 5552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.110960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35409 (local address 127.18.172.65:55491)
0924 20:24:04.111745 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.111795 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.113128 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.113622 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.113628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.113711 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:04.114205 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.114217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.114983 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.114986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.116004 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.116013 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.116525 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.116529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.116690 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.117313 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:04.117314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:04.117353 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:04.117413 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:04.117695 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:04.117706 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:04.117727 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.118249 (+ 522us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.118503 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":132,"threads_started":1}
I20260924 20:24:04.120590 5550 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35409
I20260924 20:24:04.120828 5550 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:04.121344 5550 heartbeater.cc:511] Master 127.18.172.126:35409 requested a full tablet report, sending...
I20260924 20:24:04.123014 5362 ts_manager.cc:194] Registered new tserver with Master: 6eee4b94383b4fafa7e257e65bda7bf6 (127.18.172.65:39777)
I20260924 20:24:04.124758 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.65:55491
W20260924 20:24:04.233945 5553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:04.234211 5553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:04.234251 5553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:04.238480 5553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:04.238555 5553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:04.238615 5553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:24:04.243829 5553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:04.244942 5553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:04.246317 5553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:04.256335 5561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.256340 5558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.256374 5559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:04.256584 5553 server_base.cc:1034] running on GCE node
I20260924 20:24:04.257006 5553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:04.257709 5553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:04.258940 5553 hybrid_clock.cc:648] HybridClock initialized: now 1790281444258916 us; error 58 us; skew 500 ppm
I20260924 20:24:04.260772 5553 webserver.cc:460] Webserver started at http://127.18.172.66:43441/ using document root <none> and password file <none>
I20260924 20:24:04.261381 5553 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:04.261443 5553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:04.261606 5553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:04.263885 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "88fe61b75fd74b14a0424aac8ef45e72"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "1eadfbbbbb42a04301b81ae77c44de01"
server_key_iv: "696ee31da3fbd41fabf391f8504927b7"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.264412 5553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "88fe61b75fd74b14a0424aac8ef45e72"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "1eadfbbbbb42a04301b81ae77c44de01"
server_key_iv: "696ee31da3fbd41fabf391f8504927b7"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.268191 5553 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:24:04.270642 5567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.271996 5553 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:04.272128 5553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "88fe61b75fd74b14a0424aac8ef45e72"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "1eadfbbbbb42a04301b81ae77c44de01"
server_key_iv: "696ee31da3fbd41fabf391f8504927b7"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.272243 5553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:04.283629 5553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:04.284343 5553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:04.284549 5553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:04.285161 5553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:04.286114 5553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:04.286163 5553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.286235 5553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:04.286280 5553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.295459 5553 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:36825
I20260924 20:24:04.295506 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:36825 every 8 connection(s)
I20260924 20:24:04.296394 5553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:24:04.304363 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5553
I20260924 20:24:04.304540 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:24:04.304831 19121 external_mini_cluster.cc:1428] Setting key 3487d19191688a692b9230cd566ef42b
I20260924 20:24:04.306260 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.298312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57129 (local address 127.18.172.126:35409)
0924 20:24:04.298472 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.298476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.299603 (+ 1127us) server_negotiation.cc:407] Connection header received
0924 20:24:04.301045 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.301049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.301105 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.301186 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:04.302160 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.302700 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.303880 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.304171 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.304281 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.305744 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:04.305747 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:04.305749 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:04.305760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:04.305778 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:04.305785 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:04.305868 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.306059 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.306120 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 20:24:04.306849 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--builtin_ntp_servers=127.18.172.84:35305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:04.307410 5683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.298756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35409 (local address 127.18.172.66:57129)
0924 20:24:04.299450 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.299493 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.300830 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.301355 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.301361 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.301457 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:04.302027 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.302041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.302835 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.302838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.303762 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.303771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.304421 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.304425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.304692 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.305385 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:04.305387 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:04.305446 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:04.305566 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:04.305869 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:04.305881 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:04.305889 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.306049 (+ 160us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.306340 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":115,"threads_started":1}
I20260924 20:24:04.308645 5681 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35409
I20260924 20:24:04.308872 5681 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:04.309406 5681 heartbeater.cc:511] Master 127.18.172.126:35409 requested a full tablet report, sending...
I20260924 20:24:04.310451 5362 ts_manager.cc:194] Registered new tserver with Master: 88fe61b75fd74b14a0424aac8ef45e72 (127.18.172.66:36825)
I20260924 20:24:04.311347 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.66:57129
W20260924 20:24:04.418531 5684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:04.418798 5684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:04.418836 5684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:04.422789 5684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:04.422844 5684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:04.422899 5684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:24:04.427533 5684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:35409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:04.428503 5684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:04.429804 5684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:24:04.439769 5684 server_base.cc:1034] running on GCE node
W20260924 20:24:04.439662 5692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.439662 5690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:04.439721 5689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:04.440325 5684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:04.440896 5684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:04.442055 5684 hybrid_clock.cc:648] HybridClock initialized: now 1790281444442031 us; error 34 us; skew 500 ppm
I20260924 20:24:04.443933 5684 webserver.cc:460] Webserver started at http://127.18.172.67:37353/ using document root <none> and password file <none>
I20260924 20:24:04.444499 5684 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:04.444558 5684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:04.444723 5684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:04.446734 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "6d5b646cf44a46f9a744ccbd696441b2"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "d70b84f987b76959cb5860c80df87b86"
server_key_iv: "a2345d4fe6a2e9d2ba89be43a961f699"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.447327 5684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "6d5b646cf44a46f9a744ccbd696441b2"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "d70b84f987b76959cb5860c80df87b86"
server_key_iv: "a2345d4fe6a2e9d2ba89be43a961f699"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.450901 5684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260924 20:24:04.453416 5698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.454636 5684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:24:04.454771 5684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "6d5b646cf44a46f9a744ccbd696441b2"
format_stamp: "Formatted at 2026-09-24 20:24:04 on dist-test-slave-m1qq"
server_key: "d70b84f987b76959cb5860c80df87b86"
server_key_iv: "a2345d4fe6a2e9d2ba89be43a961f699"
server_key_version: "encryptionkey@0"
I20260924 20:24:04.454885 5684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:04.488361 5684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:04.489099 5684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:04.489305 5684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:04.489919 5684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:04.490921 5684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:04.491000 5684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.491093 5684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:04.491142 5684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:04.500509 5684 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:39967
I20260924 20:24:04.500540 5811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:39967 every 8 connection(s)
I20260924 20:24:04.501461 5684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:24:04.503301 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5684
I20260924 20:24:04.503405 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:24:04.503728 19121 external_mini_cluster.cc:1428] Setting key fd21aed3ad9d4373e1724ae227d251ac
I20260924 20:24:04.511837 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.503442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:34667 (local address 127.18.172.126:35409)
0924 20:24:04.503762 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.503768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.505384 (+ 1616us) server_negotiation.cc:407] Connection header received
0924 20:24:04.507337 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.507340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.507397 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.507486 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:04.508347 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.508910 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.510007 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.510212 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.510287 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.511440 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:04.511443 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:04.511446 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:04.511460 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:04.511477 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:04.511484 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:04.511541 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.511672 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.511719 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260924 20:24:04.512715 5814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.503954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:35409 (local address 127.18.172.67:34667)
0924 20:24:04.505248 (+ 1294us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.505304 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.507107 (+ 1803us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.507664 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.507670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.507749 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:04.508243 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.508256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.509018 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.509020 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.509894 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.509903 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.510337 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.510340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.510497 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.511207 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:04.511208 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:04.511247 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:04.511311 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:04.511555 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:04.511561 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:04.511565 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.511657 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.511875 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":194,"threads_started":1}
I20260924 20:24:04.513859 5812 heartbeater.cc:347] Connected to a master server at 127.18.172.126:35409
I20260924 20:24:04.514120 5812 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:04.514637 5812 heartbeater.cc:511] Master 127.18.172.126:35409 requested a full tablet report, sending...
I20260924 20:24:04.515681 5362 ts_manager.cc:194] Registered new tserver with Master: 6d5b646cf44a46f9a744ccbd696441b2 (127.18.172.67:39967)
I20260924 20:24:04.516266 5362 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.67:34667
I20260924 20:24:04.518239 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:24:04.524188 5401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.520274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47530 (local address 127.18.172.126:35409)
0924 20:24:04.520447 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.520452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.520585 (+ 133us) server_negotiation.cc:407] Connection header received
0924 20:24:04.520735 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.520738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.520787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.520871 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:04.521148 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.521661 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.522487 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.522702 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.522809 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.523706 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:04.523709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:04.523714 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:04.523731 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:04.523749 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:04.523756 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:04.523883 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.524002 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.524038 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 20:24:04.527104 5362 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:24:04.530108 5362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:24:04.544183 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.538591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.18.172.65:39777)
0924 20:24:04.539695 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.539709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.539733 (+ 24us) server_negotiation.cc:407] Connection header received
0924 20:24:04.539811 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.539816 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.540012 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.540194 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:04.540462 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.541212 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.542892 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.543485 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.543609 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.543809 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.543914 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.544022 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":993,"thread_start_us":130,"threads_started":1}
I20260924 20:24:04.544178 5823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.538472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:39777 (local address 127.0.0.1:52108)
0924 20:24:04.538912 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.538957 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.539125 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.540192 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.540198 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.540220 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:04.540352 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.540358 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.541329 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.541334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.542745 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.542757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.543607 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.543610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.543718 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.543756 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.543871 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.544011 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":47,"thread_start_us":80,"threads_started":1}
I20260924 20:24:04.544482 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.539570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38288 (local address 127.18.172.66:36825)
0924 20:24:04.539847 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.539858 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.539879 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:24:04.539985 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.539990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.540171 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.540610 (+ 439us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:04.540719 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.541421 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.542779 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.543618 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.544021 (+ 403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.544196 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.544239 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.544325 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260924 20:24:04.544747 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.539179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36825 (local address 127.0.0.1:38288)
0924 20:24:04.539513 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.539526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.539610 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.540329 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.540332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.540349 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:04.540475 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.540486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.541535 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.541541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.542657 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.542666 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.543749 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.543752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.543835 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.543858 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.543901 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.544599 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":594,"thread_start_us":72,"threads_started":1}
I20260924 20:24:04.545650 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.538590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:39967 (local address 127.0.0.1:50598)
0924 20:24:04.539363 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.539377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.539452 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.540356 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.540358 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.540374 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:04.541436 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.541442 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.542819 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.542822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.543943 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.543952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.545236 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.545239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.545330 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.545372 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.545451 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.545516 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":942,"thread_start_us":91,"threads_started":1}
I20260924 20:24:04.546137 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.539128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50598 (local address 127.18.172.67:39967)
0924 20:24:04.539964 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.539974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.540004 (+ 30us) server_negotiation.cc:407] Connection header received
0924 20:24:04.540068 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.540072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.540213 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.540348 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:04.541570 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.542726 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.544071 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.544913 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.545015 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.545176 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.545890 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.545986 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":121,"threads_started":1}
I20260924 20:24:04.547708 5484 tablet_service.cc:1467] Processing CreateTablet for tablet 021c7a39c846415ab6043d081206b008 (DEFAULT_TABLE table=test-table [id=2e0defd786c44ae39c37af67ef5613a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:04.547724 5615 tablet_service.cc:1467] Processing CreateTablet for tablet 021c7a39c846415ab6043d081206b008 (DEFAULT_TABLE table=test-table [id=2e0defd786c44ae39c37af67ef5613a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:04.548672 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021c7a39c846415ab6043d081206b008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:04.548692 5615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021c7a39c846415ab6043d081206b008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:04.548645 5746 tablet_service.cc:1467] Processing CreateTablet for tablet 021c7a39c846415ab6043d081206b008 (DEFAULT_TABLE table=test-table [id=2e0defd786c44ae39c37af67ef5613a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:04.549515 5746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 021c7a39c846415ab6043d081206b008. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:04.555133 5829 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Bootstrap starting.
I20260924 20:24:04.556234 5830 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Bootstrap starting.
I20260924 20:24:04.556821 5831 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Bootstrap starting.
I20260924 20:24:04.557704 5829 tablet_bootstrap.cc:654] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:04.558452 5830 tablet_bootstrap.cc:654] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:04.558794 5831 tablet_bootstrap.cc:654] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:04.559428 5830 log.cc:824] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:04.559553 5831 log.cc:824] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:04.559701 5829 log.cc:824] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:04.561419 5831 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: No bootstrap required, opened a new log
I20260924 20:24:04.561487 5829 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: No bootstrap required, opened a new log
I20260924 20:24:04.561481 5830 tablet_bootstrap.cc:492] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: No bootstrap required, opened a new log
I20260924 20:24:04.561632 5831 ts_tablet_manager.cc:1397] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260924 20:24:04.561661 5829 ts_tablet_manager.cc:1397] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260924 20:24:04.561661 5830 ts_tablet_manager.cc:1397] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:24:04.565042 5831 raft_consensus.cc:348] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.565286 5831 raft_consensus.cc:374] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:04.565339 5831 raft_consensus.cc:729] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d5b646cf44a46f9a744ccbd696441b2, State: Initialized, Role: FOLLOWER
I20260924 20:24:04.565788 5831 consensus_queue.cc:260] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.565930 5830 raft_consensus.cc:348] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.565930 5829 raft_consensus.cc:348] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.566182 5830 raft_consensus.cc:374] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:04.566182 5829 raft_consensus.cc:374] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:04.566249 5830 raft_consensus.cc:729] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fe61b75fd74b14a0424aac8ef45e72, State: Initialized, Role: FOLLOWER
I20260924 20:24:04.566304 5829 raft_consensus.cc:729] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eee4b94383b4fafa7e257e65bda7bf6, State: Initialized, Role: FOLLOWER
I20260924 20:24:04.566632 5812 heartbeater.cc:503] Master 127.18.172.126:35409 was elected leader, sending a full tablet report...
I20260924 20:24:04.566731 5829 consensus_queue.cc:260] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.566903 5831 ts_tablet_manager.cc:1428] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260924 20:24:04.567838 5550 heartbeater.cc:503] Master 127.18.172.126:35409 was elected leader, sending a full tablet report...
I20260924 20:24:04.568113 5830 consensus_queue.cc:260] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.567828 5829 ts_tablet_manager.cc:1428] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260924 20:24:04.569105 5830 ts_tablet_manager.cc:1428] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:24:04.569424 5681 heartbeater.cc:503] Master 127.18.172.126:35409 was elected leader, sending a full tablet report...
W20260924 20:24:04.610020 5551 tablet.cc:2369] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:04.709538 5836 raft_consensus.cc:482] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:04.709791 5836 raft_consensus.cc:504] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.711206 5836 leader_election.cc:288] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d5b646cf44a46f9a744ccbd696441b2 (127.18.172.67:39967), 88fe61b75fd74b14a0424aac8ef45e72 (127.18.172.66:36825)
I20260924 20:24:04.715886 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.712168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:44837 (local address 127.18.172.66:36825)
0924 20:24:04.712310 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.712317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.712493 (+ 176us) server_negotiation.cc:407] Connection header received
0924 20:24:04.712549 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.712552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.712599 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.712682 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:04.712999 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.713493 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.714853 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.715299 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.715392 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.715461 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.715666 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.715746 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 20:24:04.715941 5827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.711727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:34317 (local address 127.18.172.67:39967)
0924 20:24:04.711973 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.711978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.712128 (+ 150us) server_negotiation.cc:407] Connection header received
0924 20:24:04.712355 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.712359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.712428 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.712509 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:04.712924 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.713486 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.714707 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.715372 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.715489 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.715541 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.715727 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.715819 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 20:24:04.715920 5839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.712036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:36825 (local address 127.18.172.65:44837)
0924 20:24:04.712360 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.712374 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.712473 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.712709 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.712711 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.712727 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:04.712872 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.712882 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.713616 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.713620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.714563 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.714573 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.715421 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.715428 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.715530 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.715558 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.715631 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.715719 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":46,"thread_start_us":72,"threads_started":1}
I20260924 20:24:04.715919 5838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.711601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:39967 (local address 127.18.172.65:34317)
0924 20:24:04.712006 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.712020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:04.712171 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:04.712544 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:04.712547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:04.712567 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:04.712709 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.712716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.713611 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.713616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.714564 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:04.714573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.715504 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.715507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:04.715588 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.715604 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 20:24:04.715698 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 20:24:04.715760 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":88,"thread_start_us":119,"threads_started":1}
I20260924 20:24:04.717036 5635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021c7a39c846415ab6043d081206b008" candidate_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fe61b75fd74b14a0424aac8ef45e72" is_pre_election: true
I20260924 20:24:04.717032 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021c7a39c846415ab6043d081206b008" candidate_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5b646cf44a46f9a744ccbd696441b2" is_pre_election: true
I20260924 20:24:04.717363 5635 raft_consensus.cc:2454] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eee4b94383b4fafa7e257e65bda7bf6 in term 0.
I20260924 20:24:04.717379 5766 raft_consensus.cc:2454] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eee4b94383b4fafa7e257e65bda7bf6 in term 0.
I20260924 20:24:04.717950 5438 leader_election.cc:302] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eee4b94383b4fafa7e257e65bda7bf6, 88fe61b75fd74b14a0424aac8ef45e72; no voters:
I20260924 20:24:04.718263 5836 raft_consensus.cc:2781] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:04.718344 5836 raft_consensus.cc:482] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:04.718374 5836 raft_consensus.cc:3037] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:04.719640 5836 raft_consensus.cc:504] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.720067 5836 leader_election.cc:288] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [CANDIDATE]: Term 1 election: Requested vote from peers 6d5b646cf44a46f9a744ccbd696441b2 (127.18.172.67:39967), 88fe61b75fd74b14a0424aac8ef45e72 (127.18.172.66:36825)
I20260924 20:24:04.720449 5766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021c7a39c846415ab6043d081206b008" candidate_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d5b646cf44a46f9a744ccbd696441b2"
I20260924 20:24:04.720479 5635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "021c7a39c846415ab6043d081206b008" candidate_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88fe61b75fd74b14a0424aac8ef45e72"
I20260924 20:24:04.720594 5766 raft_consensus.cc:3037] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:04.720633 5635 raft_consensus.cc:3037] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:04.721671 5635 raft_consensus.cc:2454] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eee4b94383b4fafa7e257e65bda7bf6 in term 1.
I20260924 20:24:04.721699 5766 raft_consensus.cc:2454] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eee4b94383b4fafa7e257e65bda7bf6 in term 1.
I20260924 20:24:04.721983 5438 leader_election.cc:302] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eee4b94383b4fafa7e257e65bda7bf6, 88fe61b75fd74b14a0424aac8ef45e72; no voters:
I20260924 20:24:04.722173 5836 raft_consensus.cc:2781] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:04.722448 5836 raft_consensus.cc:686] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 1 LEADER]: Becoming Leader. State: Replica: 6eee4b94383b4fafa7e257e65bda7bf6, State: Running, Role: LEADER
I20260924 20:24:04.722790 5836 consensus_queue.cc:237] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } }
I20260924 20:24:04.726491 5360 catalog_manager.cc:5495] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6eee4b94383b4fafa7e257e65bda7bf6 (127.18.172.65). New cstate: current_term: 1 leader_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6eee4b94383b4fafa7e257e65bda7bf6" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 39777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 } health_report { overall_health: UNKNOWN } } }
W20260924 20:24:04.752492 5813 tablet.cc:2369] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:04.780927 5825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.777361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52116 (local address 127.18.172.65:39777)
0924 20:24:04.777533 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:04.777537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:04.777551 (+ 14us) server_negotiation.cc:407] Connection header received
0924 20:24:04.777625 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:04.777627 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:04.777688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:04.777795 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:04.778011 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.778583 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.779482 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:04.779708 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:04.779796 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:04.780047 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:04.780473 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:04.780648 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:04.780768 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 20:24:04.780804 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 20:24:04.787134 5766 raft_consensus.cc:1261] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 1 FOLLOWER]: Refusing update from remote peer 6eee4b94383b4fafa7e257e65bda7bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:04.787153 5635 raft_consensus.cc:1261] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 1 FOLLOWER]: Refusing update from remote peer 6eee4b94383b4fafa7e257e65bda7bf6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:04.787844 5840 consensus_queue.cc:1035] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d5b646cf44a46f9a744ccbd696441b2" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 39967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:04.788092 5836 consensus_queue.cc:1035] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88fe61b75fd74b14a0424aac8ef45e72" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 36825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:04.795629 5845 mvcc.cc:205] Tried to move back new op lower bound from 7332992797840162816 to 7332992797587329024. Current Snapshot: MvccSnapshot[applied={T|T < 7332992797840162816}]
I20260924 20:24:04.804176 5360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:04.804364 5360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:04.806772 5360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45c77cf09bb140bc8e307c11485c15e2: Sending DeleteTablet for 3 replicas of tablet 021c7a39c846415ab6043d081206b008
I20260924 20:24:04.807741 5615 tablet_service.cc:1514] Processing DeleteTablet for tablet 021c7a39c846415ab6043d081206b008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:04 UTC) from {username='slave'} at 127.0.0.1:38288
I20260924 20:24:04.807754 5746 tablet_service.cc:1514] Processing DeleteTablet for tablet 021c7a39c846415ab6043d081206b008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:04 UTC) from {username='slave'} at 127.0.0.1:50598
I20260924 20:24:04.807808 5484 tablet_service.cc:1514] Processing DeleteTablet for tablet 021c7a39c846415ab6043d081206b008 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:04 UTC) from {username='slave'} at 127.0.0.1:52108
I20260924 20:24:04.808415 5854 tablet_replica.cc:330] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: stopping tablet replica
I20260924 20:24:04.808615 5855 tablet_replica.cc:330] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6: stopping tablet replica
I20260924 20:24:04.808799 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5422
I20260924 20:24:04.808781 5854 raft_consensus.cc:2229] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:04.808911 5855 raft_consensus.cc:2229] T 021c7a39c846415ab6043d081206b008 P 6eee4b94383b4fafa7e257e65bda7bf6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:24:04.809593 5854 raft_consensus.cc:2258] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:04.811125 5854 ts_tablet_manager.cc:1905] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:04.811239 5853 tablet_replica.cc:330] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: stopping tablet replica
I20260924 20:24:04.811563 5853 raft_consensus.cc:2229] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:04.811888 5853 raft_consensus.cc:2258] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:04.813514 5854 ts_tablet_manager.cc:1918] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:04.813573 5853 ts_tablet_manager.cc:1905] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:04.813627 5854 log.cc:1196] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/021c7a39c846415ab6043d081206b008
I20260924 20:24:04.813905 5854 ts_tablet_manager.cc:1939] T 021c7a39c846415ab6043d081206b008 P 88fe61b75fd74b14a0424aac8ef45e72: Deleting consensus metadata
I20260924 20:24:04.814978 5347 catalog_manager.cc:4841] TS 88fe61b75fd74b14a0424aac8ef45e72 (127.18.172.66:36825): tablet 021c7a39c846415ab6043d081206b008 (table test-table [id=2e0defd786c44ae39c37af67ef5613a5]) successfully deleted
I20260924 20:24:04.817252 5853 ts_tablet_manager.cc:1918] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:04.817364 5853 log.cc:1196] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/021c7a39c846415ab6043d081206b008
W20260924 20:24:04.817622 5348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:39777 (error 108)
I20260924 20:24:04.817699 5853 ts_tablet_manager.cc:1939] T 021c7a39c846415ab6043d081206b008 P 6d5b646cf44a46f9a744ccbd696441b2: Deleting consensus metadata
I20260924 20:24:04.818550 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5553
I20260924 20:24:04.819309 5824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:04.818370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:39777 (local address 127.0.0.1:52132)
0924 20:24:04.819126 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:04.819207 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649}
I20260924 20:24:04.819790 5347 catalog_manager.cc:4841] TS 6d5b646cf44a46f9a744ccbd696441b2 (127.18.172.67:39967): tablet 021c7a39c846415ab6043d081206b008 (table test-table [id=2e0defd786c44ae39c37af67ef5613a5]) successfully deleted
W20260924 20:24:04.823163 5348 catalog_manager.cc:4584] TS 6eee4b94383b4fafa7e257e65bda7bf6 (127.18.172.65:39777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 021c7a39c846415ab6043d081206b008: Network error: Client connection negotiation failed: client connection to 127.18.172.65:39777: connect: Connection refused (error 111)
I20260924 20:24:04.825021 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5684
I20260924 20:24:04.830878 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5331
2026-09-24T20:24:04Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1204 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:04 dist-test-slave-m1qq krb5kdc[5859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:04 dist-test-slave-m1qq krb5kdc[5859](info): set up 2 sockets
Sep 24 20:24:04 dist-test-slave-m1qq krb5kdc[5859](info): commencing operation
krb5kdc: starting...
W20260924 20:24:05.888124 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:05 dist-test-slave-m1qq krb5kdc[5859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:06.014434 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:33799
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:33799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:06.125736 5872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:06.126041 5872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:06.126120 5872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:06.130172 5872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:06.130244 5872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:06.130281 5872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:06.130336 5872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:06.130369 5872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:06.135592 5872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:33799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:33799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:06.136649 5872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:06.137682 5872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:06.147157 5877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.147363 5872 server_base.cc:1034] running on GCE node
W20260924 20:24:06.147236 5878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:06.147177 5880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.148116 5872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:06.149276 5872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:06.150465 5872 hybrid_clock.cc:648] HybridClock initialized: now 1790281446150449 us; error 29 us; skew 500 ppm
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:06.156839 5872 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:06.157840 5872 webserver.cc:460] Webserver started at http://127.18.172.126:37883/ using document root <none> and password file <none>
I20260924 20:24:06.158387 5872 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:06.158447 5872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:06.158620 5872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:06.160910 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "3d5479af3ed040fe8ab579d172b51c4e"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "f597687b1e0bbe72059dfca3877940dd"
server_key_iv: "0aaa865d3e0a50a22abcc489ecc98ff0"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.161442 5872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "3d5479af3ed040fe8ab579d172b51c4e"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "f597687b1e0bbe72059dfca3877940dd"
server_key_iv: "0aaa865d3e0a50a22abcc489ecc98ff0"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.165225 5872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:24:06.167765 5887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.169096 5872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:06.169232 5872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "3d5479af3ed040fe8ab579d172b51c4e"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "f597687b1e0bbe72059dfca3877940dd"
server_key_iv: "0aaa865d3e0a50a22abcc489ecc98ff0"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.169343 5872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:06.185119 5872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:06.188632 5872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:06.188850 5872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:06.196547 5872 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:33799
I20260924 20:24:06.196570 5939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:33799 every 8 connection(s)
I20260924 20:24:06.197525 5872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:06.200376 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:06.200937 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5872
I20260924 20:24:06.201088 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:06.201395 19121 external_mini_cluster.cc:1428] Setting key dfbd4251342194582fb7d689ad536af7
I20260924 20:24:06.207383 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: Bootstrap starting.
I20260924 20:24:06.209988 5940 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:06.210724 5940 log.cc:824] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:06.212860 5940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: No bootstrap required, opened a new log
I20260924 20:24:06.216279 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.202846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.18.172.126:33799)
0924 20:24:06.203330 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:06.203349 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:06.203396 (+ 47us) server_negotiation.cc:407] Connection header received
0924 20:24:06.204240 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:06.204264 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:06.204699 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:06.205095 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:06.205737 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.207257 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.208149 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.208868 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.209039 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:06.211693 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:06.211697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:06.211711 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:06.211738 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:06.213456 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.213904 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.213917 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.213922 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.214060 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.214353 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.214358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.214360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.214749 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:06.214968 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:06.215242 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 20:24:06.215538 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":170,"threads_started":1}
I20260924 20:24:06.216472 5940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } }
I20260924 20:24:06.216725 5940 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:06.216813 5940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5479af3ed040fe8ab579d172b51c4e, State: Initialized, Role: FOLLOWER
I20260924 20:24:06.217240 5940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } }
I20260924 20:24:06.217393 5940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:06.217450 5940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:06.217536 5940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:06.218675 5940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } }
I20260924 20:24:06.219050 5940 leader_election.cc:302] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5479af3ed040fe8ab579d172b51c4e; no voters:
I20260924 20:24:06.219332 5940 leader_election.cc:288] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:06.219445 5945 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:06.219693 5945 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [term 1 LEADER]: Becoming Leader. State: Replica: 3d5479af3ed040fe8ab579d172b51c4e, State: Running, Role: LEADER
I20260924 20:24:06.220011 5945 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } }
I20260924 20:24:06.220743 5940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:06.221745 5947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5479af3ed040fe8ab579d172b51c4e. Latest consensus state: current_term: 1 leader_uuid: "3d5479af3ed040fe8ab579d172b51c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } } }
I20260924 20:24:06.222070 5947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:06.221781 5946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5479af3ed040fe8ab579d172b51c4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5479af3ed040fe8ab579d172b51c4e" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33799 } } }
I20260924 20:24:06.222435 5946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:06.222826 5954 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:06.226094 5954 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:06.232484 5954 catalog_manager.cc:1347] Generated new cluster ID: a1359d9c0d1e404ba00d5a511281639c
I20260924 20:24:06.232563 5954 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:06.281570 5954 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:06.282914 5954 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:06.292701 5954 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: Generated new TSK 0
I20260924 20:24:06.293560 5954 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.18.172.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
Entry for principal kudu/127.18.172.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab.
I20260924 20:24:06.330272 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:06.448994 5966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:06.449258 5966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:06.449302 5966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:06.453495 5966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:06.453559 5966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:06.453666 5966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:24:06.459167 5966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.65.keytab
--principal=kudu/127.18.172.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:06.460232 5966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:06.461269 5966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:06.471647 5974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:06.471633 5972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.471697 5966 server_base.cc:1034] running on GCE node
W20260924 20:24:06.471634 5971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.472241 5966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:06.472859 5966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:06.474109 5966 hybrid_clock.cc:648] HybridClock initialized: now 1790281446474070 us; error 47 us; skew 500 ppm
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:06.476981 5966 init.cc:377] Logged in from keytab as kudu/127.18.172.65@KRBTEST.COM (short username kudu)
I20260924 20:24:06.477896 5966 webserver.cc:460] Webserver started at http://127.18.172.65:44735/ using document root <none> and password file <none>
I20260924 20:24:06.478487 5966 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:06.478554 5966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:06.478799 5966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:06.480960 5966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "433920f49af14a559b43f7d86c62bbb2"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "dc04c56b23a3cb2488b99a457f17b55b"
server_key_iv: "dd5965d748cb9651e5cf98c5484eabdd"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.481559 5966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "433920f49af14a559b43f7d86c62bbb2"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "dc04c56b23a3cb2488b99a457f17b55b"
server_key_iv: "dd5965d748cb9651e5cf98c5484eabdd"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.485603 5966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:24:06.488243 5981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.489624 5966 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:06.489764 5966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "433920f49af14a559b43f7d86c62bbb2"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "dc04c56b23a3cb2488b99a457f17b55b"
server_key_iv: "dd5965d748cb9651e5cf98c5484eabdd"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.489883 5966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:06.514098 5966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:06.517573 5966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:06.517795 5966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:06.518400 5966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:06.519312 5966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:06.519382 5966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.519469 5966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:06.519505 5966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.528570 5966 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:33883
I20260924 20:24:06.528591 6094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:33883 every 8 connection(s)
I20260924 20:24:06.529458 5966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:24:06.537247 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 5966
I20260924 20:24:06.537402 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:24:06.537725 19121 external_mini_cluster.cc:1428] Setting key f62eef410989e10ea293b06f553d9f71
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.65@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:06.543882 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.531304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:55927 (local address 127.18.172.126:33799)
0924 20:24:06.531442 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:06.531445 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:06.532278 (+ 833us) server_negotiation.cc:407] Connection header received
0924 20:24:06.533710 (+ 1432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:06.533713 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:06.533775 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:06.533877 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:06.535059 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.535591 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.536753 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.537052 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.537163 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:06.539939 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:06.539945 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:06.539952 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:06.539987 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:06.541567 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.542209 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.542211 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.542213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.542289 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.542668 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.542669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.542670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.542837 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:06.542912 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:06.543646 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 20:24:06.543759 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:24:06.545212 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.531691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:33799 (local address 127.18.172.65:55927)
0924 20:24:06.532127 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:06.532171 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:06.533476 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:06.534059 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:06.534065 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:06.534412 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:06.534882 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.534895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.535724 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.535732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.536614 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.536623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.537206 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.537209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.537421 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:06.538138 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:06.538159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:06.539773 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:06.541685 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:06.541692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:06.541712 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:06.542110 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:06.542381 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:06.542387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:06.542392 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:06.542578 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:06.542940 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:06.542949 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:06.543351 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 20:24:06.543590 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 20:24:06.543981 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":112,"threads_started":1}
I20260924 20:24:06.546594 6095 heartbeater.cc:347] Connected to a master server at 127.18.172.126:33799
I20260924 20:24:06.546847 6095 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:06.547410 6095 heartbeater.cc:511] Master 127.18.172.126:33799 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.18.172.66@KRBTEST.COM; defaulting to no policy
I20260924 20:24:06.549058 5904 ts_manager.cc:194] Registered new tserver with Master: 433920f49af14a559b43f7d86c62bbb2 (127.18.172.65:33883)
I20260924 20:24:06.550297 5904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.65@KRBTEST.COM'} at 127.18.172.65:55927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
Entry for principal kudu/127.18.172.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab.
I20260924 20:24:06.566797 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:06.678162 6100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:06.678463 6100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:06.678550 6100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:06.682651 6100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:06.682749 6100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:06.682839 6100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:24:06.688388 6100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.66.keytab
--principal=kudu/127.18.172.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:06.689679 6100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:06.690732 6100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:06.700492 6108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:06.700492 6106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.700677 6100 server_base.cc:1034] running on GCE node
W20260924 20:24:06.700785 6105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.701342 6100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:06.701965 6100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:06.703104 6100 hybrid_clock.cc:648] HybridClock initialized: now 1790281446703069 us; error 46 us; skew 500 ppm
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:06.705821 6100 init.cc:377] Logged in from keytab as kudu/127.18.172.66@KRBTEST.COM (short username kudu)
I20260924 20:24:06.706648 6100 webserver.cc:460] Webserver started at http://127.18.172.66:38289/ using document root <none> and password file <none>
I20260924 20:24:06.707261 6100 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:06.707324 6100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:06.707507 6100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:06.709481 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "aa0a114cf5444445adc158fe032d08c3"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "27b90266a9d385a025d9f17b4fc91bb0"
server_key_iv: "3a06b7fe73f64849dcead3eeba911537"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.709973 6100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "aa0a114cf5444445adc158fe032d08c3"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "27b90266a9d385a025d9f17b4fc91bb0"
server_key_iv: "3a06b7fe73f64849dcead3eeba911537"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.713739 6100 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:24:06.716184 6115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.717334 6100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:06.717427 6100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "aa0a114cf5444445adc158fe032d08c3"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "27b90266a9d385a025d9f17b4fc91bb0"
server_key_iv: "3a06b7fe73f64849dcead3eeba911537"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.717525 6100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:06.742908 6100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:06.746421 6100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:06.746605 6100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:06.747167 6100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:06.747965 6100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:06.748011 6100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.748052 6100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:06.748073 6100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.757017 6100 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:35497
I20260924 20:24:06.757048 6229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:35497 every 8 connection(s)
I20260924 20:24:06.757912 6100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:24:06.763360 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6100
I20260924 20:24:06.763506 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:24:06.763784 19121 external_mini_cluster.cc:1428] Setting key 0d93284c83f9af8a0ff3db5165e3319a
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.66@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:06.772887 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.759772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:48093 (local address 127.18.172.126:33799)
0924 20:24:06.759917 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:06.759920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:06.760751 (+ 831us) server_negotiation.cc:407] Connection header received
0924 20:24:06.762033 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:06.762035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:06.762093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:06.762169 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:06.763468 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.764114 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.765293 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.765510 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.765592 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:06.768692 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:06.768699 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:06.768706 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:06.768741 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:06.770575 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.771138 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.771145 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.771148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.771253 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:06.771634 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:06.771640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:06.771645 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:06.771903 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:06.772015 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:06.772526 (+ 511us) server_negotiation.cc:299] Negotiation successful
0924 20:24:06.772672 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:24:06.773695 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.760152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:33799 (local address 127.18.172.66:48093)
0924 20:24:06.760596 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:06.760639 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:06.761832 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:06.762327 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:06.762334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:06.762689 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:06.763291 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.763320 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.764258 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.764261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.765151 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.765160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.765666 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.765672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.765903 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:06.766557 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:06.766583 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:06.768494 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:06.770710 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:06.770716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:06.770730 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:06.770979 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:06.771368 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:06.771370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:06.771371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:06.771524 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:06.772007 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:06.772012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:06.772250 (+ 238us) client_negotiation.cc:769] Sending connection context
0924 20:24:06.772399 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 20:24:06.772764 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":117,"threads_started":1}
WARNING: no policy specified for kudu/127.18.172.67@KRBTEST.COM; defaulting to no policy
I20260924 20:24:06.774902 6230 heartbeater.cc:347] Connected to a master server at 127.18.172.126:33799
I20260924 20:24:06.775180 6230 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:06.775695 6230 heartbeater.cc:511] Master 127.18.172.126:33799 requested a full tablet report, sending...
I20260924 20:24:06.776723 5904 ts_manager.cc:194] Registered new tserver with Master: aa0a114cf5444445adc158fe032d08c3 (127.18.172.66:35497)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.67@KRBTEST.COM" created.
I20260924 20:24:06.777571 5904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.66@KRBTEST.COM'} at 127.18.172.66:48093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
Entry for principal kudu/127.18.172.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab.
I20260924 20:24:06.794624 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:06.909907 6235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:06.910188 6235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:06.910291 6235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:06.914345 6235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:06.914440 6235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:06.914527 6235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:24:06.919977 6235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.67.keytab
--principal=kudu/127.18.172.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:33799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:06.921094 6235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:06.922214 6235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:06.931654 6241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:06.931653 6240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:06.931622 6243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:06.932067 6235 server_base.cc:1034] running on GCE node
I20260924 20:24:06.932513 6235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:06.933199 6235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:06.934404 6235 hybrid_clock.cc:648] HybridClock initialized: now 1790281446934376 us; error 48 us; skew 500 ppm
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:06.937373 6235 init.cc:377] Logged in from keytab as kudu/127.18.172.67@KRBTEST.COM (short username kudu)
I20260924 20:24:06.938300 6235 webserver.cc:460] Webserver started at http://127.18.172.67:46523/ using document root <none> and password file <none>
I20260924 20:24:06.938977 6235 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:06.939146 6235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:06.939410 6235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:06.941684 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "c6bfea917bbe4856858c4168802b10d6"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "af8a263cadcc40aa50642750fd9bfc08"
server_key_iv: "68eccba057febd22f44e3f9376747b43"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.942296 6235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "c6bfea917bbe4856858c4168802b10d6"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "af8a263cadcc40aa50642750fd9bfc08"
server_key_iv: "68eccba057febd22f44e3f9376747b43"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.946488 6235 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:24:06.949229 6250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.950721 6235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:06.950865 6235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "c6bfea917bbe4856858c4168802b10d6"
format_stamp: "Formatted at 2026-09-24 20:24:06 on dist-test-slave-m1qq"
server_key: "af8a263cadcc40aa50642750fd9bfc08"
server_key_iv: "68eccba057febd22f44e3f9376747b43"
server_key_version: "encryptionkey@0"
I20260924 20:24:06.950989 6235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:06.975196 6235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:06.978189 6235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:06.978435 6235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:06.979094 6235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:06.980010 6235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:06.980087 6235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.980168 6235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:06.980227 6235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:06.990290 6235 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:41741
I20260924 20:24:06.990314 6363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:41741 every 8 connection(s)
I20260924 20:24:06.991355 6235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
Sep 24 20:24:06 dist-test-slave-m1qq krb5kdc[5859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281446, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.67@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:07.001272 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6235
I20260924 20:24:07.001434 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:24:07.001780 19121 external_mini_cluster.cc:1428] Setting key 85a00c1687e66a807a4e0d7ad7b1d622
I20260924 20:24:07.005667 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.993295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:58679 (local address 127.18.172.126:33799)
0924 20:24:06.993451 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:06.993455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:06.994366 (+ 911us) server_negotiation.cc:407] Connection header received
0924 20:24:06.995786 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:06.995789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:06.995835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:06.995910 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:06.997125 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.997657 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.998818 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.999150 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.999237 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.001858 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:07.001861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:07.001864 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:07.001892 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:07.003713 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:07.004320 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:07.004322 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:07.004324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:07.004398 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:07.004695 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:07.004697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:07.004698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:07.004867 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:07.004938 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.005425 (+ 487us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.005550 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 20:24:07.006626 6366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:06.993683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:33799 (local address 127.18.172.67:58679)
0924 20:24:06.994213 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:06.994254 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:06.995586 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:06.996106 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:06.996112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:06.996490 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:06.996984 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.996997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.997790 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.997793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.998683 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:06.998692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:06.999294 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:06.999297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:06.999439 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.000027 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:07.000042 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:07.001685 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:07.003847 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:07.003856 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:07.003874 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:07.004225 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:07.004479 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:07.004482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:07.004483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:07.004617 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:07.004959 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:07.004964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:07.005208 (+ 244us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.005381 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.005646 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":118,"threads_started":1}
I20260924 20:24:07.007732 6364 heartbeater.cc:347] Connected to a master server at 127.18.172.126:33799
I20260924 20:24:07.007968 6364 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:07.008422 6364 heartbeater.cc:511] Master 127.18.172.126:33799 requested a full tablet report, sending...
I20260924 20:24:07.009421 5904 ts_manager.cc:194] Registered new tserver with Master: c6bfea917bbe4856858c4168802b10d6 (127.18.172.67:41741)
I20260924 20:24:07.009992 5904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.18.172.67@KRBTEST.COM'} at 127.18.172.67:58679
I20260924 20:24:07.015849 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:24:07.025326 5943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.017794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48612 (local address 127.18.172.126:33799)
0924 20:24:07.017982 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.017986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.018081 (+ 95us) server_negotiation.cc:407] Connection header received
0924 20:24:07.018170 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.018172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.018213 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.018274 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:07.018896 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.019448 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.020274 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.020483 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.020560 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.022179 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:07.022182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:07.022184 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:07.022207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:07.023828 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:07.024232 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:07.024235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:07.024236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:07.024311 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:07.024532 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:07.024534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:07.024535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:07.024740 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:07.024840 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.025029 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.025127 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 20:24:07.028467 5904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:24:07.031558 5904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:24:07.046023 6375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.039795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:35497 (local address 127.0.0.1:45190)
0924 20:24:07.040858 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.040911 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.041043 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.041627 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.041633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.041659 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:07.041793 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.041799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.043275 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.043279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.044323 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.044338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045460 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.045580 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.045622 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.045725 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.045850 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260924 20:24:07.046200 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.040009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.18.172.66:35497)
0924 20:24:07.040310 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.040322 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.041181 (+ 859us) server_negotiation.cc:407] Connection header received
0924 20:24:07.041259 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.041263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.041454 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.041616 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:07.041925 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.043129 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.044453 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045325 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045452 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.045606 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.045948 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.046050 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":126,"threads_started":1}
I20260924 20:24:07.046727 6378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.040457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:41741 (local address 127.0.0.1:57440)
0924 20:24:07.041362 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.041372 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.041431 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.041828 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.041831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.041850 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:07.041946 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.041955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.042918 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.042924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.044456 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.044468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045705 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045708 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.045835 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.046458 (+ 623us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.046541 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.046608 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":633,"thread_start_us":89,"threads_started":1}
I20260924 20:24:07.046815 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.040777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.18.172.67:41741)
0924 20:24:07.041128 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.041141 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.041450 (+ 309us) server_negotiation.cc:407] Connection header received
0924 20:24:07.041530 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.041538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.041705 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.041858 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:07.042072 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.042790 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.044590 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045381 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045492 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.045606 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.046581 (+ 975us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.046683 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":143,"threads_started":1}
I20260924 20:24:07.048830 6164 tablet_service.cc:1467] Processing CreateTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 (DEFAULT_TABLE table=test-table [id=9b282c5d038a42c1aa633b47dcdffd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:07.049501 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.039750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33883 (local address 127.0.0.1:32934)
0924 20:24:07.041152 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.041164 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.041238 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.042093 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.042096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.042113 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:07.042219 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.042225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.043727 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.043730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.044827 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.044834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045723 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.045864 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.045890 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.046155 (+ 265us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.046246 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":76,"thread_start_us":75,"threads_started":1}
I20260924 20:24:07.049942 6164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b75bc9b075f4b55946de66cd7ff5023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:07.049974 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.040451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.18.172.65:33883)
0924 20:24:07.041645 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.041659 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.041680 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:24:07.041751 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.041756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.041935 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.042359 (+ 424us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:07.042486 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.043585 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.044941 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.045608 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.045728 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.045860 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.049535 (+ 3675us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.049835 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1074,"thread_start_us":115,"threads_started":1}
I20260924 20:24:07.051935 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 (DEFAULT_TABLE table=test-table [id=9b282c5d038a42c1aa633b47dcdffd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:07.052474 6029 tablet_service.cc:1467] Processing CreateTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 (DEFAULT_TABLE table=test-table [id=9b282c5d038a42c1aa633b47dcdffd01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:07.052860 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b75bc9b075f4b55946de66cd7ff5023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:07.053406 6029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b75bc9b075f4b55946de66cd7ff5023. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:07.057302 6381 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: Bootstrap starting.
I20260924 20:24:07.059278 6382 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Bootstrap starting.
I20260924 20:24:07.059774 6381 tablet_bootstrap.cc:654] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:07.060386 6383 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: Bootstrap starting.
I20260924 20:24:07.061291 6381 log.cc:824] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:07.061401 6382 tablet_bootstrap.cc:654] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:07.062387 6383 tablet_bootstrap.cc:654] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:07.062664 6382 log.cc:824] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:07.063202 6383 log.cc:824] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:07.063555 6381 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: No bootstrap required, opened a new log
I20260924 20:24:07.063755 6381 ts_tablet_manager.cc:1397] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260924 20:24:07.064517 6382 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: No bootstrap required, opened a new log
I20260924 20:24:07.064711 6382 ts_tablet_manager.cc:1397] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 20:24:07.065168 6383 tablet_bootstrap.cc:492] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: No bootstrap required, opened a new log
I20260924 20:24:07.065340 6383 ts_tablet_manager.cc:1397] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:24:07.067751 6382 raft_consensus.cc:348] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.068039 6382 raft_consensus.cc:374] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:07.067889 6381 raft_consensus.cc:348] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.068096 6382 raft_consensus.cc:729] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bfea917bbe4856858c4168802b10d6, State: Initialized, Role: FOLLOWER
I20260924 20:24:07.068140 6381 raft_consensus.cc:374] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:07.068186 6381 raft_consensus.cc:729] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0a114cf5444445adc158fe032d08c3, State: Initialized, Role: FOLLOWER
I20260924 20:24:07.068593 6382 consensus_queue.cc:260] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.068837 6381 consensus_queue.cc:260] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.069545 6230 heartbeater.cc:503] Master 127.18.172.126:33799 was elected leader, sending a full tablet report...
I20260924 20:24:07.069747 6382 ts_tablet_manager.cc:1428] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260924 20:24:07.069829 6381 ts_tablet_manager.cc:1428] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260924 20:24:07.070075 6364 heartbeater.cc:503] Master 127.18.172.126:33799 was elected leader, sending a full tablet report...
I20260924 20:24:07.069826 6383 raft_consensus.cc:348] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.071086 6383 raft_consensus.cc:374] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:07.071151 6383 raft_consensus.cc:729] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 433920f49af14a559b43f7d86c62bbb2, State: Initialized, Role: FOLLOWER
I20260924 20:24:07.071573 6383 consensus_queue.cc:260] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.072228 6095 heartbeater.cc:503] Master 127.18.172.126:33799 was elected leader, sending a full tablet report...
I20260924 20:24:07.072661 6383 ts_tablet_manager.cc:1428] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260924 20:24:07.133464 6388 raft_consensus.cc:482] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:07.133704 6388 raft_consensus.cc:504] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.134948 6388 leader_election.cc:288] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 433920f49af14a559b43f7d86c62bbb2 (127.18.172.65:33883), c6bfea917bbe4856858c4168802b10d6 (127.18.172.67:41741)
I20260924 20:24:07.138572 6379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.135358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:57501 (local address 127.18.172.65:33883)
0924 20:24:07.135502 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.135509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.135524 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:24:07.135579 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.135584 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.135639 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.135709 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:07.136070 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.136605 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.137628 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.138031 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.138160 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.138215 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.138385 (+ 170us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.138454 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 20:24:07.138572 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.135180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33883 (local address 127.18.172.66:57501)
0924 20:24:07.135329 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.135345 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.135437 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.135795 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.135800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.135822 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:07.135944 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.135949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.136719 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.136724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.137508 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.137516 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.138149 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.138153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.138254 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.138282 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.138368 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.138438 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 20:24:07.139209 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.135435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:41741 (local address 127.18.172.66:47547)
0924 20:24:07.135936 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.135947 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.136069 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.136369 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.136372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.136389 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:07.136521 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.136527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.137216 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.137221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.138016 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.138025 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.138800 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.138805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.138900 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.138928 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.139044 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.139105 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"mutex_wait_us":10,"thread_start_us":152,"threads_started":1}
I20260924 20:24:07.139250 6380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.135699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:47547 (local address 127.18.172.67:41741)
0924 20:24:07.135830 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.135837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.136029 (+ 192us) server_negotiation.cc:407] Connection header received
0924 20:24:07.136197 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.136200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.136243 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.136368 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:07.136626 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.137103 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.138139 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.138589 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.138731 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.138789 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.139061 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.139136 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 20:24:07.139284 6049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b75bc9b075f4b55946de66cd7ff5023" candidate_uuid: "aa0a114cf5444445adc158fe032d08c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433920f49af14a559b43f7d86c62bbb2" is_pre_election: true
I20260924 20:24:07.139595 6049 raft_consensus.cc:2454] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa0a114cf5444445adc158fe032d08c3 in term 0.
I20260924 20:24:07.139936 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b75bc9b075f4b55946de66cd7ff5023" candidate_uuid: "aa0a114cf5444445adc158fe032d08c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bfea917bbe4856858c4168802b10d6" is_pre_election: true
I20260924 20:24:07.140121 6119 leader_election.cc:302] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 433920f49af14a559b43f7d86c62bbb2, aa0a114cf5444445adc158fe032d08c3; no voters:
I20260924 20:24:07.140208 6318 raft_consensus.cc:2454] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa0a114cf5444445adc158fe032d08c3 in term 0.
I20260924 20:24:07.140388 6388 raft_consensus.cc:2781] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:07.140476 6388 raft_consensus.cc:482] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:07.140527 6388 raft_consensus.cc:3037] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:07.141592 6388 raft_consensus.cc:504] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.141911 6388 leader_election.cc:288] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [CANDIDATE]: Term 1 election: Requested vote from peers 433920f49af14a559b43f7d86c62bbb2 (127.18.172.65:33883), c6bfea917bbe4856858c4168802b10d6 (127.18.172.67:41741)
I20260924 20:24:07.142220 6049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b75bc9b075f4b55946de66cd7ff5023" candidate_uuid: "aa0a114cf5444445adc158fe032d08c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "433920f49af14a559b43f7d86c62bbb2"
I20260924 20:24:07.142273 6318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b75bc9b075f4b55946de66cd7ff5023" candidate_uuid: "aa0a114cf5444445adc158fe032d08c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6bfea917bbe4856858c4168802b10d6"
I20260924 20:24:07.142351 6049 raft_consensus.cc:3037] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:07.142413 6318 raft_consensus.cc:3037] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:07.143469 6049 raft_consensus.cc:2454] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0a114cf5444445adc158fe032d08c3 in term 1.
I20260924 20:24:07.143494 6318 raft_consensus.cc:2454] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0a114cf5444445adc158fe032d08c3 in term 1.
I20260924 20:24:07.143784 6119 leader_election.cc:302] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa0a114cf5444445adc158fe032d08c3, c6bfea917bbe4856858c4168802b10d6; no voters:
I20260924 20:24:07.143985 6388 raft_consensus.cc:2781] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:07.144209 6388 raft_consensus.cc:686] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 1 LEADER]: Becoming Leader. State: Replica: aa0a114cf5444445adc158fe032d08c3, State: Running, Role: LEADER
I20260924 20:24:07.144532 6388 consensus_queue.cc:237] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } }
I20260924 20:24:07.147702 5903 catalog_manager.cc:5495] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa0a114cf5444445adc158fe032d08c3 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "aa0a114cf5444445adc158fe032d08c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0a114cf5444445adc158fe032d08c3" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 } health_report { overall_health: UNKNOWN } } }
I20260924 20:24:07.168052 6377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.164371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.18.172.66:35497)
0924 20:24:07.164615 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.164622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.164637 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:24:07.164726 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.164729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.164784 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.164900 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:07.165176 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.165700 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.166540 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.166762 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.166869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.167232 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:07.167650 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:07.167785 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.167831 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.167887 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260924 20:24:07.174006 6049 raft_consensus.cc:1261] T 3b75bc9b075f4b55946de66cd7ff5023 P 433920f49af14a559b43f7d86c62bbb2 [term 1 FOLLOWER]: Refusing update from remote peer aa0a114cf5444445adc158fe032d08c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:07.174041 6318 raft_consensus.cc:1261] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 1 FOLLOWER]: Refusing update from remote peer aa0a114cf5444445adc158fe032d08c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:07.174860 6388 consensus_queue.cc:1035] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6bfea917bbe4856858c4168802b10d6" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:07.175110 6391 consensus_queue.cc:1035] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "433920f49af14a559b43f7d86c62bbb2" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:07.183362 6394 mvcc.cc:205] Tried to move back new op lower bound from 7332992807616819200 to 7332992807506722816. Current Snapshot: MvccSnapshot[applied={T|T < 7332992807616819200}]
I20260924 20:24:07.183714 6396 mvcc.cc:205] Tried to move back new op lower bound from 7332992807616819200 to 7332992807506722816. Current Snapshot: MvccSnapshot[applied={T|T < 7332992807616819200}]
I20260924 20:24:07.184955 6395 mvcc.cc:205] Tried to move back new op lower bound from 7332992807616819200 to 7332992807506722816. Current Snapshot: MvccSnapshot[applied={T|T < 7332992807616819200}]
I20260924 20:24:07.192229 5902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:07.192414 5902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:07.194944 5902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d5479af3ed040fe8ab579d172b51c4e: Sending DeleteTablet for 3 replicas of tablet 3b75bc9b075f4b55946de66cd7ff5023
I20260924 20:24:07.195812 6164 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:45190
I20260924 20:24:07.195977 6029 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:32934
I20260924 20:24:07.196345 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5966
I20260924 20:24:07.196322 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b75bc9b075f4b55946de66cd7ff5023 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:07 UTC) from {username='kudu', principal='kudu/127.18.172.126@KRBTEST.COM'} at 127.0.0.1:57440
I20260924 20:24:07.197256 6406 tablet_replica.cc:330] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: stopping tablet replica
I20260924 20:24:07.197569 6406 raft_consensus.cc:2229] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:07.197863 6406 raft_consensus.cc:2258] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:07.198231 6403 tablet_replica.cc:330] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3: stopping tablet replica
I20260924 20:24:07.198460 6403 raft_consensus.cc:2229] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:24:07.198812 6403 raft_consensus.cc:2258] T 3b75bc9b075f4b55946de66cd7ff5023 P aa0a114cf5444445adc158fe032d08c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:07.204095 6406 ts_tablet_manager.cc:1905] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 20:24:07.204499 5890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:33883 (error 108)
I20260924 20:24:07.205837 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6100
I20260924 20:24:07.205890 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.205607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:33883 (local address 127.0.0.1:32940)
0924 20:24:07.205735 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.205809 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 20:24:07.207261 6406 ts_tablet_manager.cc:1918] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:07.207365 6406 log.cc:1196] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/3b75bc9b075f4b55946de66cd7ff5023
I20260924 20:24:07.207706 6406 ts_tablet_manager.cc:1939] T 3b75bc9b075f4b55946de66cd7ff5023 P c6bfea917bbe4856858c4168802b10d6: Deleting consensus metadata
W20260924 20:24:07.208920 5890 catalog_manager.cc:4584] TS 433920f49af14a559b43f7d86c62bbb2 (127.18.172.65:33883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b75bc9b075f4b55946de66cd7ff5023: Network error: Client connection negotiation failed: client connection to 127.18.172.65:33883: connect: Connection refused (error 111)
I20260924 20:24:07.209134 5890 catalog_manager.cc:4841] TS c6bfea917bbe4856858c4168802b10d6 (127.18.172.67:41741): tablet 3b75bc9b075f4b55946de66cd7ff5023 (table test-table [id=9b282c5d038a42c1aa633b47dcdffd01]) successfully deleted
I20260924 20:24:07.211161 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6235
I20260924 20:24:07.211655 6376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.211409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:35497 (local address 127.0.0.1:45218)
0924 20:24:07.211507 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.211571 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.66:35497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 20:24:07.216462 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 5872
2026-09-24T20:24:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2386 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T20:24:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:07Z Disabled control of system clock
I20260924 20:24:07.247277 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38905
--webserver_interface=127.18.172.126
--webserver_port=0
--builtin_ntp_servers=127.18.172.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:07.359853 6411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:07.360146 6411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:07.360183 6411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:07.364315 6411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:07.364379 6411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:07.364396 6411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:07.364413 6411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:07.364430 6411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:07.369969 6411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:38905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:07.371132 6411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:07.372604 6411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:07.381178 6417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:07.381172 6416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:07.381172 6419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:07.381264 6411 server_base.cc:1034] running on GCE node
I20260924 20:24:07.381976 6411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:07.383206 6411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:07.384368 6411 hybrid_clock.cc:648] HybridClock initialized: now 1790281447384331 us; error 52 us; skew 500 ppm
I20260924 20:24:07.386210 6411 webserver.cc:460] Webserver started at http://127.18.172.126:41535/ using document root <none> and password file <none>
I20260924 20:24:07.386794 6411 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:07.386859 6411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:07.387108 6411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:07.389205 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "70e3e18678df40fa8fa14ffd5ee289d0"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "69ae926adfcf75973a7a89a519b1eac0"
server_key_iv: "e6963c468446889cd83738cef7186352"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.389815 6411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "70e3e18678df40fa8fa14ffd5ee289d0"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "69ae926adfcf75973a7a89a519b1eac0"
server_key_iv: "e6963c468446889cd83738cef7186352"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.393669 6411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:24:07.396164 6425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.397369 6411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:07.397500 6411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "70e3e18678df40fa8fa14ffd5ee289d0"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "69ae926adfcf75973a7a89a519b1eac0"
server_key_iv: "e6963c468446889cd83738cef7186352"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.397626 6411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:07.409552 6411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:07.410220 6411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:07.410426 6411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:07.417644 6411 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:38905
I20260924 20:24:07.417651 6477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:38905 every 8 connection(s)
I20260924 20:24:07.418601 6411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:07.421737 6478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:07.423266 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6411
I20260924 20:24:07.423414 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:07.423694 19121 external_mini_cluster.cc:1428] Setting key 4384b840f5e55fbd1050a38f339bc0ea
I20260924 20:24:07.428342 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: Bootstrap starting.
I20260924 20:24:07.430823 6478 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:07.431731 6478 log.cc:824] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:07.433732 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.424974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53006 (local address 127.18.172.126:38905)
0924 20:24:07.425617 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.425637 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.425676 (+ 39us) server_negotiation.cc:407] Connection header received
0924 20:24:07.426540 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.426563 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.427070 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.427439 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:07.427852 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.429219 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.430068 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.430734 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.430896 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.431803 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:07.431810 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:07.431827 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:07.432283 (+ 456us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:07.432340 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:07.432360 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:07.432469 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.432611 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.432864 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":96,"threads_started":1}
I20260924 20:24:07.433861 6478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: No bootstrap required, opened a new log
I20260924 20:24:07.437690 6478 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } }
I20260924 20:24:07.437911 6478 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:07.437945 6478 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e3e18678df40fa8fa14ffd5ee289d0, State: Initialized, Role: FOLLOWER
I20260924 20:24:07.438333 6478 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } }
I20260924 20:24:07.438447 6478 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:07.438490 6478 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:07.438556 6478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:07.439437 6478 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } }
I20260924 20:24:07.439714 6478 leader_election.cc:302] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70e3e18678df40fa8fa14ffd5ee289d0; no voters:
I20260924 20:24:07.439956 6478 leader_election.cc:288] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:07.440099 6483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:07.440336 6483 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [term 1 LEADER]: Becoming Leader. State: Replica: 70e3e18678df40fa8fa14ffd5ee289d0, State: Running, Role: LEADER
I20260924 20:24:07.440634 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } }
I20260924 20:24:07.441129 6478 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:07.442556 6485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70e3e18678df40fa8fa14ffd5ee289d0. Latest consensus state: current_term: 1 leader_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } } }
I20260924 20:24:07.442680 6485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:07.443153 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70e3e18678df40fa8fa14ffd5ee289d0" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38905 } } }
I20260924 20:24:07.443249 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:07.443543 6492 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:07.446259 6492 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:07.452004 6492 catalog_manager.cc:1347] Generated new cluster ID: 0a3ed17aa1ae4ab4a09686a231eb0b64
I20260924 20:24:07.452102 6492 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:07.463727 6492 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:07.464924 6492 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:07.471050 6492 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: Generated new TSK 0
I20260924 20:24:07.471692 6492 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:07.476639 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.65:0
--local_ip_for_outbound_sockets=127.18.172.65
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--builtin_ntp_servers=127.18.172.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 20:24:07.596292 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:07.596534 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:07.596603 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:07.601226 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:07.601284 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:07.601384 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.65
I20260924 20:24:07.606189 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.18.172.65
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.65
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:07.607295 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:07.608620 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:07.618376 6510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:07.618366 6507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:07.618366 6508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:07.618618 6502 server_base.cc:1034] running on GCE node
I20260924 20:24:07.619176 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:07.619827 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:07.621019 6502 hybrid_clock.cc:648] HybridClock initialized: now 1790281447620985 us; error 42 us; skew 500 ppm
I20260924 20:24:07.622817 6502 webserver.cc:460] Webserver started at http://127.18.172.65:43489/ using document root <none> and password file <none>
I20260924 20:24:07.623533 6502 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:07.623629 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:07.623839 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:07.625869 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data/instance:
uuid: "ceb2d0c18a6747fea84f6a289b94e282"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "f82a3f44f88639b091ddefcc36ec75bc"
server_key_iv: "d5b72f76027382cbea682fe9d08634fe"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.626418 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance:
uuid: "ceb2d0c18a6747fea84f6a289b94e282"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "f82a3f44f88639b091ddefcc36ec75bc"
server_key_iv: "d5b72f76027382cbea682fe9d08634fe"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.630046 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:24:07.632469 6516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.633809 6502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:07.633939 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal
uuid: "ceb2d0c18a6747fea84f6a289b94e282"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "f82a3f44f88639b091ddefcc36ec75bc"
server_key_iv: "d5b72f76027382cbea682fe9d08634fe"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.634050 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:07.654381 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:07.655153 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:07.655356 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:07.655938 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:07.656884 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:07.656955 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.657025 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:07.657076 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.666190 6502 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.65:38517
I20260924 20:24:07.666218 6629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.65:38517 every 8 connection(s)
I20260924 20:24:07.667142 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/data/info.pb
I20260924 20:24:07.674350 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6502
I20260924 20:24:07.674497 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-0/wal/instance
I20260924 20:24:07.674783 19121 external_mini_cluster.cc:1428] Setting key d200156ed2ac139abbf7c5e61cc65f96
I20260924 20:24:07.676851 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.66:0
--local_ip_for_outbound_sockets=127.18.172.66
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--builtin_ntp_servers=127.18.172.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:07.677165 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.669279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.65:49963 (local address 127.18.172.126:38905)
0924 20:24:07.669418 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.669422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.670532 (+ 1110us) server_negotiation.cc:407] Connection header received
0924 20:24:07.671971 (+ 1439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.671974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.672040 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.672121 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:07.673098 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.673637 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.675035 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.675323 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.675444 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.676593 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:07.676599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:07.676605 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:07.676622 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:07.676647 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:07.676661 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:07.676744 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.676892 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.676960 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 20:24:07.678040 6632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.669619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38905 (local address 127.18.172.65:49963)
0924 20:24:07.670370 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.670422 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.671763 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.672273 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.672280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.672362 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:07.672947 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.672963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.673776 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.673779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.674868 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.674886 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.675493 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.675497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.675663 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.676327 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:07.676328 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:07.676369 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:07.676432 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:07.676751 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:07.676758 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:07.676762 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.676854 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.677190 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":162,"threads_started":1}
I20260924 20:24:07.679301 6630 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38905
I20260924 20:24:07.679544 6630 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:07.680132 6630 heartbeater.cc:511] Master 127.18.172.126:38905 requested a full tablet report, sending...
I20260924 20:24:07.681743 6442 ts_manager.cc:194] Registered new tserver with Master: ceb2d0c18a6747fea84f6a289b94e282 (127.18.172.65:38517)
I20260924 20:24:07.682920 6442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.65:49963
W20260924 20:24:07.788821 6633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:07.789091 6633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:07.789129 6633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:07.793215 6633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:07.793277 6633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:07.793332 6633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.66
I20260924 20:24:07.798156 6633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.18.172.66
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.66
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:07.799235 6633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:07.800594 6633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:07.810505 6639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:07.810562 6633 server_base.cc:1034] running on GCE node
W20260924 20:24:07.810513 6641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:07.810539 6638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:07.811219 6633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:07.811885 6633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:07.813068 6633 hybrid_clock.cc:648] HybridClock initialized: now 1790281447813033 us; error 46 us; skew 500 ppm
I20260924 20:24:07.815092 6633 webserver.cc:460] Webserver started at http://127.18.172.66:43531/ using document root <none> and password file <none>
I20260924 20:24:07.815676 6633 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:07.815766 6633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:07.815995 6633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:07.818046 6633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data/instance:
uuid: "cf50fb3903184cff8e267c1e32135738"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "93a2eac6c20e9468343d874d7d3d303b"
server_key_iv: "c5d20818e36f7a7b28d57fdbe3699280"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.818606 6633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance:
uuid: "cf50fb3903184cff8e267c1e32135738"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "93a2eac6c20e9468343d874d7d3d303b"
server_key_iv: "c5d20818e36f7a7b28d57fdbe3699280"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.822347 6633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260924 20:24:07.824852 6647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.826068 6633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:24:07.826205 6633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal
uuid: "cf50fb3903184cff8e267c1e32135738"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "93a2eac6c20e9468343d874d7d3d303b"
server_key_iv: "c5d20818e36f7a7b28d57fdbe3699280"
server_key_version: "encryptionkey@0"
I20260924 20:24:07.826318 6633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:07.844907 6633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:07.845624 6633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:07.845824 6633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:07.846457 6633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:07.847635 6633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:07.847705 6633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.847770 6633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:07.847820 6633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:07.857110 6633 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.66:35539
I20260924 20:24:07.857129 6760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.66:35539 every 8 connection(s)
I20260924 20:24:07.858038 6633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/data/info.pb
I20260924 20:24:07.863727 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6633
I20260924 20:24:07.863858 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal/instance
I20260924 20:24:07.864192 19121 external_mini_cluster.cc:1428] Setting key b988c0ece824be421e17ad6757171a11
I20260924 20:24:07.866312 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.18.172.67:0
--local_ip_for_outbound_sockets=127.18.172.67
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--builtin_ntp_servers=127.18.172.84:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 20:24:07.867494 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.859802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:52995 (local address 127.18.172.126:38905)
0924 20:24:07.859932 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:07.859936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:07.860924 (+ 988us) server_negotiation.cc:407] Connection header received
0924 20:24:07.862238 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:07.862241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:07.862288 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:07.862363 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:07.863212 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.863935 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.865224 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.865547 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.865654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.866877 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:07.866884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:07.866890 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:07.866907 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:07.866931 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:07.866949 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:07.867058 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:07.867219 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 20:24:07.867288 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:24:07.868361 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:07.860149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38905 (local address 127.18.172.66:52995)
0924 20:24:07.860768 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:07.860808 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:07.862057 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:07.862526 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:07.862532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:07.862611 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:07.863094 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.863106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.864061 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.864064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.865104 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:07.865120 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:07.865706 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:07.865710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:07.865889 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:07.866620 (+ 731us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:07.866622 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:07.866664 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:07.866727 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:07.867079 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:07.867086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:07.867091 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 20:24:07.867192 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 20:24:07.867412 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
I20260924 20:24:07.869524 6761 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38905
I20260924 20:24:07.869750 6761 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:07.870433 6761 heartbeater.cc:511] Master 127.18.172.126:38905 requested a full tablet report, sending...
I20260924 20:24:07.871565 6442 ts_manager.cc:194] Registered new tserver with Master: cf50fb3903184cff8e267c1e32135738 (127.18.172.66:35539)
I20260924 20:24:07.872157 6442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.66:52995
W20260924 20:24:07.977926 6764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:07.978194 6764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:07.978258 6764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:07.982609 6764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:07.982668 6764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:07.982754 6764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.18.172.67
I20260924 20:24:07.987709 6764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.18.172.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.18.172.67
--webserver_port=0
--tserver_master_addrs=127.18.172.126:38905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.18.172.67
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:07.988798 6764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:07.990185 6764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:08.000087 6772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:08.000111 6769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:08.000082 6770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:08.000386 6764 server_base.cc:1034] running on GCE node
I20260924 20:24:08.000926 6764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:08.001557 6764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:08.002739 6764 hybrid_clock.cc:648] HybridClock initialized: now 1790281448002701 us; error 56 us; skew 500 ppm
I20260924 20:24:08.004596 6764 webserver.cc:460] Webserver started at http://127.18.172.67:43171/ using document root <none> and password file <none>
I20260924 20:24:08.005178 6764 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:08.005266 6764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:08.005489 6764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:08.007570 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data/instance:
uuid: "e9015463781340c4ab5714c54ee44470"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "db488ac37e6b2f8055ca65d7ad302314"
server_key_iv: "ab1e00b5e867e713c7f0ca570c7e9c02"
server_key_version: "encryptionkey@0"
I20260924 20:24:08.008131 6764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance:
uuid: "e9015463781340c4ab5714c54ee44470"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "db488ac37e6b2f8055ca65d7ad302314"
server_key_iv: "ab1e00b5e867e713c7f0ca570c7e9c02"
server_key_version: "encryptionkey@0"
I20260924 20:24:08.011754 6764 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:24:08.014219 6778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:08.015563 6764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260924 20:24:08.015689 6764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal
uuid: "e9015463781340c4ab5714c54ee44470"
format_stamp: "Formatted at 2026-09-24 20:24:07 on dist-test-slave-m1qq"
server_key: "db488ac37e6b2f8055ca65d7ad302314"
server_key_iv: "ab1e00b5e867e713c7f0ca570c7e9c02"
server_key_version: "encryptionkey@0"
I20260924 20:24:08.015800 6764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:08.031507 6764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:08.032138 6764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:08.032327 6764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:08.032951 6764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 20:24:08.033886 6764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 20:24:08.033954 6764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:08.034024 6764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 20:24:08.034078 6764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:08.043473 6764 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.67:37257
I20260924 20:24:08.043501 6891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.67:37257 every 8 connection(s)
I20260924 20:24:08.044375 6764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/data/info.pb
I20260924 20:24:08.053079 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6764
I20260924 20:24:08.053223 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal/instance
I20260924 20:24:08.053401 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.046221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.67:45901 (local address 127.18.172.126:38905)
0924 20:24:08.046396 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.046401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.047606 (+ 1205us) server_negotiation.cc:407] Connection header received
0924 20:24:08.048912 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.048915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.048963 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.049044 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:08.049903 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.050436 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.051597 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.051807 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.051885 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.052904 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:08.052906 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:08.052908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:08.052921 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:08.052939 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:08.052947 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:08.053003 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.053209 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.053278 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 20:24:08.053519 19121 external_mini_cluster.cc:1428] Setting key f162a0e9544105aa7fe04ffd871a093e
I20260924 20:24:08.054441 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.046651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:38905 (local address 127.18.172.67:45901)
0924 20:24:08.047426 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.047467 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.048716 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.049200 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.049206 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.049289 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:08.049781 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.049792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.050547 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.050550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.051491 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.051500 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.051943 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.051945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.052080 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.052626 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 20:24:08.052627 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:08.052665 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 20:24:08.052726 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:08.053030 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:08.053039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:08.053046 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.053182 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.053478 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":107,"threads_started":1}
I20260924 20:24:08.055495 6892 heartbeater.cc:347] Connected to a master server at 127.18.172.126:38905
I20260924 20:24:08.055747 6892 heartbeater.cc:464] Registering TS with master...
I20260924 20:24:08.056273 6892 heartbeater.cc:511] Master 127.18.172.126:38905 requested a full tablet report, sending...
I20260924 20:24:08.057200 6442 ts_manager.cc:194] Registered new tserver with Master: e9015463781340c4ab5714c54ee44470 (127.18.172.67:37257)
I20260924 20:24:08.057744 6442 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.18.172.67:45901
I20260924 20:24:08.067389 19121 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 20:24:08.072871 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.069336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53010 (local address 127.18.172.126:38905)
0924 20:24:08.069466 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.069470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.069600 (+ 130us) server_negotiation.cc:407] Connection header received
0924 20:24:08.069708 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.069710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.069747 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.069813 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:08.070116 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.070730 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.071583 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.071785 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.071862 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.072559 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:08.072562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:08.072564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:08.072574 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:08.072590 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:08.072597 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:08.072654 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.072724 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.072769 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:24:08.075774 6442 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 20:24:08.078796 6442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 20:24:08.093303 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.088085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:38517 (local address 127.0.0.1:43516)
0924 20:24:08.088604 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.088620 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.088712 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.089379 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.089382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.089400 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:08.089483 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.089489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090502 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.090508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.091830 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.091840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.092814 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.092816 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.092897 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.092926 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.093013 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.093140 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":152,"thread_start_us":98,"threads_started":1}
I20260924 20:24:08.093326 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.088517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43516 (local address 127.18.172.65:38517)
0924 20:24:08.088925 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.088937 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.089002 (+ 65us) server_negotiation.cc:407] Connection header received
0924 20:24:08.089072 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.089077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.089251 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.089415 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:08.089587 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090392 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.091962 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.092698 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.092812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.093010 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.093065 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.093158 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":121,"threads_started":1}
I20260924 20:24:08.093576 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.087452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.18.172.67:37257)
0924 20:24:08.087901 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.087915 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.088318 (+ 403us) server_negotiation.cc:407] Connection header received
0924 20:24:08.088639 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.088647 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.088863 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.089024 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:08.089335 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090550 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.092246 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.092993 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.093101 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.093260 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.093306 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.093396 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":105,"threads_started":1}
I20260924 20:24:08.093870 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.087338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37257 (local address 127.0.0.1:50998)
0924 20:24:08.088166 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.088218 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.088442 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.089043 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.089050 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.089074 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:08.089207 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.089213 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090678 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.090682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.092131 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.092137 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.093112 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.093115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.093198 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.093219 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.093276 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.093339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":71,"spinlock_wait_cycles":6656,"thread_start_us":97,"threads_started":1}
I20260924 20:24:08.095471 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.087698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.66:35539 (local address 127.0.0.1:56256)
0924 20:24:08.088305 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.088318 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.088442 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.089343 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.089346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.089364 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:08.089465 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.089471 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090785 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.090787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.092119 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.092131 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.095073 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.095077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.095158 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.095175 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.095267 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.095335 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":98,"spinlock_wait_cycles":10752,"thread_start_us":82,"threads_started":1}
I20260924 20:24:08.095566 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.088162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56256 (local address 127.18.172.66:35539)
0924 20:24:08.088794 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.088805 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.088825 (+ 20us) server_negotiation.cc:407] Connection header received
0924 20:24:08.088896 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.088900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.089064 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.089194 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:08.089601 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.090680 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.093962 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.094717 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.094826 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.095032 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.095301 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.095404 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":106,"threads_started":1}
I20260924 20:24:08.096750 6826 tablet_service.cc:1467] Processing CreateTablet for tablet 004b9a9b90224963ad17e6290982f973 (DEFAULT_TABLE table=test-table [id=66a5877f74874037a86345fafcd62b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:08.096750 6564 tablet_service.cc:1467] Processing CreateTablet for tablet 004b9a9b90224963ad17e6290982f973 (DEFAULT_TABLE table=test-table [id=66a5877f74874037a86345fafcd62b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:08.097888 6564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004b9a9b90224963ad17e6290982f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:08.097888 6826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004b9a9b90224963ad17e6290982f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:08.097880 6695 tablet_service.cc:1467] Processing CreateTablet for tablet 004b9a9b90224963ad17e6290982f973 (DEFAULT_TABLE table=test-table [id=66a5877f74874037a86345fafcd62b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 20:24:08.098981 6695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004b9a9b90224963ad17e6290982f973. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:08.104952 6909 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Bootstrap starting.
I20260924 20:24:08.105973 6911 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Bootstrap starting.
I20260924 20:24:08.107151 6909 tablet_bootstrap.cc:654] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:08.107327 6910 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Bootstrap starting.
I20260924 20:24:08.107777 6911 tablet_bootstrap.cc:654] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:08.107997 6909 log.cc:824] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:08.108584 6911 log.cc:824] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:08.109382 6910 tablet_bootstrap.cc:654] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:08.110081 6909 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: No bootstrap required, opened a new log
I20260924 20:24:08.110136 6910 log.cc:824] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:08.110313 6909 ts_tablet_manager.cc:1397] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260924 20:24:08.110576 6911 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: No bootstrap required, opened a new log
I20260924 20:24:08.110754 6911 ts_tablet_manager.cc:1397] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:24:08.112200 6910 tablet_bootstrap.cc:492] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: No bootstrap required, opened a new log
I20260924 20:24:08.112366 6910 ts_tablet_manager.cc:1397] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 20:24:08.113757 6909 raft_consensus.cc:348] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.114107 6909 raft_consensus.cc:374] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:08.114171 6909 raft_consensus.cc:729] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf50fb3903184cff8e267c1e32135738, State: Initialized, Role: FOLLOWER
I20260924 20:24:08.114625 6909 consensus_queue.cc:260] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.114915 6911 raft_consensus.cc:348] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.115307 6761 heartbeater.cc:503] Master 127.18.172.126:38905 was elected leader, sending a full tablet report...
I20260924 20:24:08.115221 6911 raft_consensus.cc:374] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:08.115738 6911 raft_consensus.cc:729] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9015463781340c4ab5714c54ee44470, State: Initialized, Role: FOLLOWER
I20260924 20:24:08.116199 6911 consensus_queue.cc:260] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.116446 6910 raft_consensus.cc:348] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.116688 6910 raft_consensus.cc:374] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:08.116740 6910 raft_consensus.cc:729] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceb2d0c18a6747fea84f6a289b94e282, State: Initialized, Role: FOLLOWER
I20260924 20:24:08.116849 6909 ts_tablet_manager.cc:1428] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260924 20:24:08.117121 6911 ts_tablet_manager.cc:1428] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260924 20:24:08.117146 6910 consensus_queue.cc:260] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.117477 6892 heartbeater.cc:503] Master 127.18.172.126:38905 was elected leader, sending a full tablet report...
I20260924 20:24:08.117812 6630 heartbeater.cc:503] Master 127.18.172.126:38905 was elected leader, sending a full tablet report...
I20260924 20:24:08.118032 6910 ts_tablet_manager.cc:1428] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260924 20:24:08.168505 6631 tablet.cc:2369] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:08.270427 6915 raft_consensus.cc:482] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 20:24:08.270825 6915 raft_consensus.cc:504] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.272221 6915 leader_election.cc:288] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9015463781340c4ab5714c54ee44470 (127.18.172.67:37257), ceb2d0c18a6747fea84f6a289b94e282 (127.18.172.65:38517)
I20260924 20:24:08.276888 6763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.272450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.67:37257 (local address 127.18.172.66:49361)
0924 20:24:08.272606 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.272624 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.272747 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.273053 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.273058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.273083 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:08.273207 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.273215 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.274353 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.274356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.275653 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.275664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.276477 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.276479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.276554 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.276575 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.276636 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.276733 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":25}
I20260924 20:24:08.276939 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.272592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:49361 (local address 127.18.172.67:37257)
0924 20:24:08.272757 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.272763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.272784 (+ 21us) server_negotiation.cc:407] Connection header received
0924 20:24:08.272849 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.272852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.272907 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.273005 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:08.273361 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.274219 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.275775 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.276349 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.276477 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.276541 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.276684 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.276817 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 20:24:08.277333 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.272852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.18.172.66:36001 (local address 127.18.172.65:38517)
0924 20:24:08.272993 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.272998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.273332 (+ 334us) server_negotiation.cc:407] Connection header received
0924 20:24:08.273399 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.273405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.273468 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.273572 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 20:24:08.273871 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.274557 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.275994 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.276601 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.276737 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.276780 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.277108 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.277198 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 20:24:08.277690 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.272730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:38517 (local address 127.18.172.66:36001)
0924 20:24:08.273213 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.273227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:08.273318 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:08.273586 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:08.273589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:08.273607 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 20:24:08.273724 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.273730 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.274672 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.274675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.275880 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:08.275887 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.276751 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.276754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:08.276953 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.276977 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 20:24:08.277477 (+ 500us) client_negotiation.cc:240] Negotiation successful
0924 20:24:08.277550 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":51,"thread_start_us":206,"threads_started":1}
I20260924 20:24:08.277645 6846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004b9a9b90224963ad17e6290982f973" candidate_uuid: "cf50fb3903184cff8e267c1e32135738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9015463781340c4ab5714c54ee44470" is_pre_election: true
I20260924 20:24:08.277977 6846 raft_consensus.cc:2454] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf50fb3903184cff8e267c1e32135738 in term 0.
I20260924 20:24:08.278396 6584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004b9a9b90224963ad17e6290982f973" candidate_uuid: "cf50fb3903184cff8e267c1e32135738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb2d0c18a6747fea84f6a289b94e282" is_pre_election: true
I20260924 20:24:08.278544 6649 leader_election.cc:302] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf50fb3903184cff8e267c1e32135738, e9015463781340c4ab5714c54ee44470; no voters:
I20260924 20:24:08.278661 6584 raft_consensus.cc:2454] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf50fb3903184cff8e267c1e32135738 in term 0.
I20260924 20:24:08.278858 6915 raft_consensus.cc:2781] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 20:24:08.278961 6915 raft_consensus.cc:482] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 20:24:08.279040 6915 raft_consensus.cc:3037] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:08.280092 6915 raft_consensus.cc:504] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.280444 6915 leader_election.cc:288] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [CANDIDATE]: Term 1 election: Requested vote from peers e9015463781340c4ab5714c54ee44470 (127.18.172.67:37257), ceb2d0c18a6747fea84f6a289b94e282 (127.18.172.65:38517)
I20260924 20:24:08.280807 6846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004b9a9b90224963ad17e6290982f973" candidate_uuid: "cf50fb3903184cff8e267c1e32135738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9015463781340c4ab5714c54ee44470"
I20260924 20:24:08.280830 6584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004b9a9b90224963ad17e6290982f973" candidate_uuid: "cf50fb3903184cff8e267c1e32135738" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ceb2d0c18a6747fea84f6a289b94e282"
I20260924 20:24:08.280939 6846 raft_consensus.cc:3037] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:08.280973 6584 raft_consensus.cc:3037] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:08.282042 6846 raft_consensus.cc:2454] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf50fb3903184cff8e267c1e32135738 in term 1.
I20260924 20:24:08.282071 6584 raft_consensus.cc:2454] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf50fb3903184cff8e267c1e32135738 in term 1.
I20260924 20:24:08.282370 6649 leader_election.cc:302] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf50fb3903184cff8e267c1e32135738, e9015463781340c4ab5714c54ee44470; no voters:
I20260924 20:24:08.282584 6915 raft_consensus.cc:2781] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:08.282827 6915 raft_consensus.cc:686] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 1 LEADER]: Becoming Leader. State: Replica: cf50fb3903184cff8e267c1e32135738, State: Running, Role: LEADER
I20260924 20:24:08.283226 6915 consensus_queue.cc:237] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } }
I20260924 20:24:08.286758 6441 catalog_manager.cc:5495] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf50fb3903184cff8e267c1e32135738 (127.18.172.66). New cstate: current_term: 1 leader_uuid: "cf50fb3903184cff8e267c1e32135738" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf50fb3903184cff8e267c1e32135738" member_type: VOTER last_known_addr { host: "127.18.172.66" port: 35539 } health_report { overall_health: HEALTHY } } }
W20260924 20:24:08.295333 6893 tablet.cc:2369] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 20:24:08.329656 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.325589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56258 (local address 127.18.172.66:35539)
0924 20:24:08.325750 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:08.325754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:08.325769 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:24:08.325937 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:08.325939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:08.326001 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:08.326146 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:08.326396 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.326897 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.328097 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:08.328343 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:08.328457 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:08.328778 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:08.329214 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:08.329383 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:08.329415 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 20:24:08.329469 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 20:24:08.336081 6584 raft_consensus.cc:1261] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 1 FOLLOWER]: Refusing update from remote peer cf50fb3903184cff8e267c1e32135738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:08.336118 6846 raft_consensus.cc:1261] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 1 FOLLOWER]: Refusing update from remote peer cf50fb3903184cff8e267c1e32135738: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:08.337023 6919 consensus_queue.cc:1035] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ceb2d0c18a6747fea84f6a289b94e282" member_type: VOTER last_known_addr { host: "127.18.172.65" port: 38517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:08.337244 6915 consensus_queue.cc:1035] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9015463781340c4ab5714c54ee44470" member_type: VOTER last_known_addr { host: "127.18.172.67" port: 37257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:08.347865 6923 mvcc.cc:205] Tried to move back new op lower bound from 7332992812375457792 to 7332992812170698752. Current Snapshot: MvccSnapshot[applied={T|T < 7332992812375457792}]
I20260924 20:24:08.355872 6441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:08.356060 6441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 20:24:08.358669 6441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70e3e18678df40fa8fa14ffd5ee289d0: Sending DeleteTablet for 3 replicas of tablet 004b9a9b90224963ad17e6290982f973
I20260924 20:24:08.359854 6826 tablet_service.cc:1514] Processing DeleteTablet for tablet 004b9a9b90224963ad17e6290982f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:08 UTC) from {username='slave'} at 127.0.0.1:50998
I20260924 20:24:08.359956 6564 tablet_service.cc:1514] Processing DeleteTablet for tablet 004b9a9b90224963ad17e6290982f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:08 UTC) from {username='slave'} at 127.0.0.1:43516
I20260924 20:24:08.360275 6695 tablet_service.cc:1514] Processing DeleteTablet for tablet 004b9a9b90224963ad17e6290982f973 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 20:24:08 UTC) from {username='slave'} at 127.0.0.1:56256
I20260924 20:24:08.360445 6933 tablet_replica.cc:330] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282: stopping tablet replica
I20260924 20:24:08.360728 6934 tablet_replica.cc:330] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: stopping tablet replica
I20260924 20:24:08.360774 6933 raft_consensus.cc:2229] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:08.361042 6934 raft_consensus.cc:2229] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 1 LEADER]: Raft consensus shutting down.
I20260924 20:24:08.361120 6933 raft_consensus.cc:2258] T 004b9a9b90224963ad17e6290982f973 P ceb2d0c18a6747fea84f6a289b94e282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:08.361756 6932 tablet_replica.cc:330] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: stopping tablet replica
I20260924 20:24:08.362058 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6502
I20260924 20:24:08.362267 6932 raft_consensus.cc:2229] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 20:24:08.362643 6932 raft_consensus.cc:2258] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:08.364648 6934 raft_consensus.cc:2258] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 20:24:08.364892 6932 ts_tablet_manager.cc:1905] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:08.366458 6934 ts_tablet_manager.cc:1905] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 20:24:08.367749 6932 ts_tablet_manager.cc:1918] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:08.367851 6932 log.cc:1196] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-2/wal/wals/004b9a9b90224963ad17e6290982f973
I20260924 20:24:08.368145 6932 ts_tablet_manager.cc:1939] T 004b9a9b90224963ad17e6290982f973 P e9015463781340c4ab5714c54ee44470: Deleting consensus metadata
I20260924 20:24:08.369436 6427 catalog_manager.cc:4841] TS e9015463781340c4ab5714c54ee44470 (127.18.172.67:37257): tablet 004b9a9b90224963ad17e6290982f973 (table test-table [id=66a5877f74874037a86345fafcd62b79]) successfully deleted
I20260924 20:24:08.370714 6934 ts_tablet_manager.cc:1918] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 20:24:08.370839 6934 log.cc:1196] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Deleting WAL directory at /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790281188136637-19121-0/minicluster-data/ts-1/wal/wals/004b9a9b90224963ad17e6290982f973
I20260924 20:24:08.371258 6934 ts_tablet_manager.cc:1939] T 004b9a9b90224963ad17e6290982f973 P cf50fb3903184cff8e267c1e32135738: Deleting consensus metadata
W20260924 20:24:08.372480 6427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.18.172.65:38517 (error 108)
I20260924 20:24:08.372649 6426 catalog_manager.cc:4841] TS cf50fb3903184cff8e267c1e32135738 (127.18.172.66:35539): tablet 004b9a9b90224963ad17e6290982f973 (table test-table [id=66a5877f74874037a86345fafcd62b79]) successfully deleted
I20260924 20:24:08.372720 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6633
I20260924 20:24:08.374565 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:08.374178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.65:38517 (local address 127.0.0.1:43520)
0924 20:24:08.374330 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:08.374434 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.65:38517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260924 20:24:08.374758 6427 catalog_manager.cc:4584] TS ceb2d0c18a6747fea84f6a289b94e282 (127.18.172.65:38517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 004b9a9b90224963ad17e6290982f973: Network error: Client connection negotiation failed: client connection to 127.18.172.65:38517: connect: Connection refused (error 111)
I20260924 20:24:08.378615 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6764
I20260924 20:24:08.384392 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6411
2026-09-24T20:24:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1167 ms)
[----------] 6 tests from EncryptionPolicyTest (10747 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:08 dist-test-slave-m1qq krb5kdc[6938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:08 dist-test-slave-m1qq krb5kdc[6938](info): set up 2 sockets
Sep 24 20:24:08 dist-test-slave-m1qq krb5kdc[6938](info): commencing operation
krb5kdc: starting...
W20260924 20:24:09.440379 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[6938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:09.570631 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:09.681887 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:09.682149 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:09.682191 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:09.686203 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:09.686275 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:09.686295 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:09.686311 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:09.686327 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:09.691774 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:09.692816 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:09.693953 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:09.703130 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:09.703168 6951 server_base.cc:1034] running on GCE node
W20260924 20:24:09.703117 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:09.703117 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:09.703868 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:09.705142 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:09.706359 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790281449706331 us; error 36 us; skew 500 ppm
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[6938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:09.709596 6951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:09.710542 6951 webserver.cc:460] Webserver started at http://127.0.0.1:38203/ using document root <none> and password file <none>
I20260924 20:24:09.711171 6951 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:09.711242 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:09.711429 6951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:09.713605 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "3d47fdbc15564e9295d4cb7b510cd08d"
format_stamp: "Formatted at 2026-09-24 20:24:09 on dist-test-slave-m1qq"
server_key: "672582e5ab93857cd495f48f7d822c93"
server_key_iv: "fb27359fb1851702a065997aaecf1f8e"
server_key_version: "encryptionkey@0"
I20260924 20:24:09.714159 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "3d47fdbc15564e9295d4cb7b510cd08d"
format_stamp: "Formatted at 2026-09-24 20:24:09 on dist-test-slave-m1qq"
server_key: "672582e5ab93857cd495f48f7d822c93"
server_key_iv: "fb27359fb1851702a065997aaecf1f8e"
server_key_version: "encryptionkey@0"
I20260924 20:24:09.718245 6951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:24:09.720964 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:09.722329 6951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:09.722506 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "3d47fdbc15564e9295d4cb7b510cd08d"
format_stamp: "Formatted at 2026-09-24 20:24:09 on dist-test-slave-m1qq"
server_key: "672582e5ab93857cd495f48f7d822c93"
server_key_iv: "fb27359fb1851702a065997aaecf1f8e"
server_key_version: "encryptionkey@0"
I20260924 20:24:09.722625 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:09.742911 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:09.746165 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:09.746337 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:09.754287 6951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35151
I20260924 20:24:09.754307 7018 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35151 every 8 connection(s)
I20260924 20:24:09.755450 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:09.757471 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 6951
I20260924 20:24:09.757620 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:09.757998 19121 external_mini_cluster.cc:1428] Setting key 4d0fa8cf81b9af56febfdea557a806b9
I20260924 20:24:09.758437 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:09.765729 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: Bootstrap starting.
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[6938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:24:09.768093 7019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:09.768977 7019 log.cc:824] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:09.770974 7019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: No bootstrap required, opened a new log
I20260924 20:24:09.773506 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:09.759532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58095 (local address 127.0.0.1:35151)
0924 20:24:09.759978 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:09.759997 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:09.760039 (+ 42us) server_negotiation.cc:407] Connection header received
0924 20:24:09.760820 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:09.760844 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:09.761296 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:09.761669 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:09.762393 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:09.763909 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:09.764844 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:09.765516 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:09.765679 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:09.768309 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:09.768314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:09.768329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:09.768357 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:09.770142 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:09.770710 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:09.770720 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:09.770725 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:09.770830 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:09.771252 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:09.771255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:09.771257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:09.771782 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:09.771998 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:09.772268 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 20:24:09.772492 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260924 20:24:09.775372 7019 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 20:24:09.775652 7019 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:09.775717 7019 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d47fdbc15564e9295d4cb7b510cd08d, State: Initialized, Role: FOLLOWER
I20260924 20:24:09.776201 7019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 20:24:09.776356 7019 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:09.776424 7019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:09.776453 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:09.776513 7019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:09.777702 7019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 20:24:09.778105 7019 leader_election.cc:302] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d47fdbc15564e9295d4cb7b510cd08d; no voters:
I20260924 20:24:09.778424 7019 leader_election.cc:288] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:09.778559 7028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:09.778844 7028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [term 1 LEADER]: Becoming Leader. State: Replica: 3d47fdbc15564e9295d4cb7b510cd08d, State: Running, Role: LEADER
I20260924 20:24:09.779206 7028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } }
I20260924 20:24:09.779870 7019 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:09.781009 7031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } } }
I20260924 20:24:09.781158 7031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:09.781469 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d47fdbc15564e9295d4cb7b510cd08d. Latest consensus state: current_term: 1 leader_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d47fdbc15564e9295d4cb7b510cd08d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35151 } } }
I20260924 20:24:09.781654 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d [sys.catalog]: This master's current role is: LEADER
W20260924 20:24:09.785514 7045 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:09.785619 7045 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:09.785742 7038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:09.786628 7038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:09.788384 7022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:09.778728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47113 (local address 127.0.0.1:35151)
0924 20:24:09.778827 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:09.778831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:09.779472 (+ 641us) server_negotiation.cc:407] Connection header received
0924 20:24:09.779747 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:09.779750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:09.779797 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:09.779886 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:09.780766 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:09.781538 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:09.782855 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:09.783185 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:09.783270 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:09.784844 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:09.784848 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:09.784851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:09.784875 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:09.786467 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:09.786995 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:09.787048 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:09.787050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:09.787161 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:09.787497 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:09.787501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:09.787503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:09.787695 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:09.787864 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:09.788131 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 20:24:09.788224 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 20:24:09.792690 7038 catalog_manager.cc:1347] Generated new cluster ID: 0e4543dc6a0046018e5147f3ea9694f3
I20260924 20:24:09.792757 7038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:09.800217 7038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:09.801327 7038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:09.811404 7038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d47fdbc15564e9295d4cb7b510cd08d: Generated new TSK 0
I20260924 20:24:09.812207 7038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:09.816457 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 6951
W20260924 20:24:09.822345 6936 connection.cc:535] client connection to 127.0.0.1:35151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35151): Connection reset by peer (error 104)
2026-09-24T20:24:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[7050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[7050](info): set up 2 sockets
Sep 24 20:24:09 dist-test-slave-m1qq krb5kdc[7050](info): commencing operation
krb5kdc: starting...
W20260924 20:24:10.896766 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:10 dist-test-slave-m1qq krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:11.026372 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:11.136423 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:11.136749 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:11.136834 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:11.140938 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:11.141047 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:11.141098 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:11.141156 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:11.141196 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:11.146692 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:11.147964 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:11.149111 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:11.158315 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:11.158241 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:11.158324 7063 server_base.cc:1034] running on GCE node
W20260924 20:24:11.158236 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:11.159245 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:11.160430 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:11.161640 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790281451161620 us; error 35 us; skew 500 ppm
Sep 24 20:24:11 dist-test-slave-m1qq krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:11.169503 7063 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:11.170534 7063 webserver.cc:460] Webserver started at http://127.0.0.1:39765/ using document root <none> and password file <none>
I20260924 20:24:11.171155 7063 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:11.171226 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:11.171401 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:11.173682 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "e5756288740a4d0ea7ff7601432873e2"
format_stamp: "Formatted at 2026-09-24 20:24:11 on dist-test-slave-m1qq"
server_key: "d35b8709a6dc412f0ec24a93182c2097"
server_key_iv: "01c32cb22de2cc0e6548a0221e6b5dac"
server_key_version: "encryptionkey@0"
I20260924 20:24:11.174216 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "e5756288740a4d0ea7ff7601432873e2"
format_stamp: "Formatted at 2026-09-24 20:24:11 on dist-test-slave-m1qq"
server_key: "d35b8709a6dc412f0ec24a93182c2097"
server_key_iv: "01c32cb22de2cc0e6548a0221e6b5dac"
server_key_version: "encryptionkey@0"
I20260924 20:24:11.178197 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:24:11.180747 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:11.182029 7063 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:11.182180 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "e5756288740a4d0ea7ff7601432873e2"
format_stamp: "Formatted at 2026-09-24 20:24:11 on dist-test-slave-m1qq"
server_key: "d35b8709a6dc412f0ec24a93182c2097"
server_key_iv: "01c32cb22de2cc0e6548a0221e6b5dac"
server_key_version: "encryptionkey@0"
I20260924 20:24:11.182300 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:11.195611 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:11.198750 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:11.198966 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:11.206476 7063 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46101
I20260924 20:24:11.206516 7130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46101 every 8 connection(s)
I20260924 20:24:11.207551 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:11.210345 7131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:11.212720 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7063
I20260924 20:24:11.212877 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:11.213167 19121 external_mini_cluster.cc:1428] Setting key f971ad238cf66b0524e860b932060abd
I20260924 20:24:11.216657 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2: Bootstrap starting.
I20260924 20:24:11.219239 7131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:11.220103 7131 log.cc:824] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:11.222337 7131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2: No bootstrap required, opened a new log
I20260924 20:24:11.222518 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:11.214639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52567 (local address 127.0.0.1:46101)
0924 20:24:11.215077 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:11.215093 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:11.215129 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:24:11.215780 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:11.215797 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:11.216126 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:11.216396 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:11.216838 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:11.217831 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:11.219126 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:11.219684 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:11.219811 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:11.220782 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:11.220787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:11.220800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:11.221179 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:11.221235 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:11.221257 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:11.221412 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:11.221618 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 20:24:11.221751 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":136,"threads_started":1}
I20260924 20:24:11.225589 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:11.226203 7131 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260924 20:24:11.226423 7131 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:11.226480 7131 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5756288740a4d0ea7ff7601432873e2, State: Initialized, Role: FOLLOWER
I20260924 20:24:11.226936 7131 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260924 20:24:11.227138 7131 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:11.227223 7131 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:11.227325 7131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:11.228600 7131 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260924 20:24:11.229115 7131 leader_election.cc:302] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5756288740a4d0ea7ff7601432873e2; no voters:
I20260924 20:24:11.229512 7131 leader_election.cc:288] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:11.229588 7141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:11.229799 7141 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [term 1 LEADER]: Becoming Leader. State: Replica: e5756288740a4d0ea7ff7601432873e2, State: Running, Role: LEADER
I20260924 20:24:11.230165 7141 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } }
I20260924 20:24:11.230993 7131 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:11.231971 7143 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5756288740a4d0ea7ff7601432873e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260924 20:24:11.232100 7143 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:11.232518 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5756288740a4d0ea7ff7601432873e2. Latest consensus state: current_term: 1 leader_uuid: "e5756288740a4d0ea7ff7601432873e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5756288740a4d0ea7ff7601432873e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46101 } } }
I20260924 20:24:11.232672 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:11.233193 7134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:11.227869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34213 (local address 127.0.0.1:46101)
0924 20:24:11.227997 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:11.228002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:11.228264 (+ 262us) server_negotiation.cc:407] Connection header received
0924 20:24:11.228435 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:11.228438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:11.228479 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:11.228553 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:11.228987 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:11.229830 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:11.230918 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:11.231335 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:11.231460 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:11.232584 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:11.232590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:11.232593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:11.232640 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:11.232669 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:11.232683 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:11.232773 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:11.232996 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 20:24:11.233034 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:24:11.233480 7147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:11.236783 7147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:11.242718 7147 catalog_manager.cc:1347] Generated new cluster ID: dab5c70a5b4a4e53b7287fc77892d5e6
I20260924 20:24:11.242806 7147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:11.271648 7147 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:11.273005 7147 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:11.279871 7147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5756288740a4d0ea7ff7601432873e2: Generated new TSK 0
I20260924 20:24:11.280735 7147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:11.289956 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7063
W20260924 20:24:11.295672 7048 connection.cc:535] client connection to 127.0.0.1:46101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46101): Connection reset by peer (error 104)
2026-09-24T20:24:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1473 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:11 dist-test-slave-m1qq krb5kdc[7162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:11 dist-test-slave-m1qq krb5kdc[7162](info): set up 2 sockets
Sep 24 20:24:11 dist-test-slave-m1qq krb5kdc[7162](info): commencing operation
krb5kdc: starting...
W20260924 20:24:12.336287 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:12 dist-test-slave-m1qq krb5kdc[7162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:12.466224 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:12.584028 7175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:12.584296 7175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:12.584337 7175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:12.588279 7175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:12.588338 7175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:12.588358 7175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:12.588375 7175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:12.588390 7175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:12.593823 7175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34403
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:12.594866 7175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:12.595876 7175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:12.605014 7180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:12.604978 7181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:12.604976 7183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:12.605290 7175 server_base.cc:1034] running on GCE node
I20260924 20:24:12.605825 7175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:12.607116 7175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:12.608317 7175 hybrid_clock.cc:648] HybridClock initialized: now 1790281452608289 us; error 43 us; skew 500 ppm
Sep 24 20:24:12 dist-test-slave-m1qq krb5kdc[7162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:12.611192 7175 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:12.612042 7175 webserver.cc:460] Webserver started at http://127.0.0.1:40993/ using document root <none> and password file <none>
I20260924 20:24:12.612602 7175 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:12.612694 7175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:12.612923 7175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:12.614951 7175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "ba658970d25b49858b5faf1acc27a312"
format_stamp: "Formatted at 2026-09-24 20:24:12 on dist-test-slave-m1qq"
server_key: "47164230fdd781e7c391b21e7c114e0e"
server_key_iv: "29ed12d4f320a38ae3a137cc35c38ed7"
server_key_version: "encryptionkey@0"
I20260924 20:24:12.615586 7175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "ba658970d25b49858b5faf1acc27a312"
format_stamp: "Formatted at 2026-09-24 20:24:12 on dist-test-slave-m1qq"
server_key: "47164230fdd781e7c391b21e7c114e0e"
server_key_iv: "29ed12d4f320a38ae3a137cc35c38ed7"
server_key_version: "encryptionkey@0"
I20260924 20:24:12.619555 7175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 20:24:12.622069 7190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:12.623560 7175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:12.623710 7175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "ba658970d25b49858b5faf1acc27a312"
format_stamp: "Formatted at 2026-09-24 20:24:12 on dist-test-slave-m1qq"
server_key: "47164230fdd781e7c391b21e7c114e0e"
server_key_iv: "29ed12d4f320a38ae3a137cc35c38ed7"
server_key_version: "encryptionkey@0"
I20260924 20:24:12.623833 7175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:12.628834 7175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:12.632018 7175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:12.632226 7175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:12.639963 7175 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34403
I20260924 20:24:12.640007 7242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34403 every 8 connection(s)
I20260924 20:24:12.641042 7175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:12.642751 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7175
I20260924 20:24:12.642926 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:12.643280 19121 external_mini_cluster.cc:1428] Setting key 6d3c681ad7fdabcde9bb9834563b6424
I20260924 20:24:12.643908 7243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:12.648480 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:12.644670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60023 (local address 127.0.0.1:34403)
0924 20:24:12.645173 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:12.645187 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:12.645222 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:24:12.645841 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:12.645856 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:12.646130 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:12.646470 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:12.647302 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:12.647308 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:12.647329 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:12.647339 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:12.647388 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:12.647412 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:12.647569 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:12.647639 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 20:24:12.647784 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":133,"threads_started":1}
I20260924 20:24:12.650651 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: Bootstrap starting.
I20260924 20:24:12.651748 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:12.653183 7243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:12.654099 7243 log.cc:824] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:12.655587 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:12.653869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35477 (local address 127.0.0.1:34403)
0924 20:24:12.653990 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:12.653995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:12.654224 (+ 229us) server_negotiation.cc:407] Connection header received
0924 20:24:12.654345 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:12.654348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:12.654379 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:12.654444 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:12.655231 (+ 787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:12.655233 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:12.655235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:12.655245 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:12.655269 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:12.655278 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:12.655346 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:12.655441 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 20:24:12.655477 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 20:24:12.656488 7243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: No bootstrap required, opened a new log
I20260924 20:24:12.660019 7243 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260924 20:24:12.660238 7243 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:12.660270 7243 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba658970d25b49858b5faf1acc27a312, State: Initialized, Role: FOLLOWER
I20260924 20:24:12.660686 7243 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260924 20:24:12.660806 7243 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:12.660868 7243 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:12.660964 7243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:12.661849 7243 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260924 20:24:12.662217 7243 leader_election.cc:302] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba658970d25b49858b5faf1acc27a312; no voters:
I20260924 20:24:12.662578 7243 leader_election.cc:288] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:12.662706 7253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:12.663132 7253 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [term 1 LEADER]: Becoming Leader. State: Replica: ba658970d25b49858b5faf1acc27a312, State: Running, Role: LEADER
I20260924 20:24:12.663457 7253 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } }
I20260924 20:24:12.663903 7243 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:12.665328 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba658970d25b49858b5faf1acc27a312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260924 20:24:12.665336 7255 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba658970d25b49858b5faf1acc27a312. Latest consensus state: current_term: 1 leader_uuid: "ba658970d25b49858b5faf1acc27a312" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba658970d25b49858b5faf1acc27a312" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34403 } } }
I20260924 20:24:12.665478 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:12.665478 7255 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312 [sys.catalog]: This master's current role is: LEADER
W20260924 20:24:12.668569 7268 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:12.668634 7268 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:12.668709 7267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:12.669498 7267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:12.675493 7267 catalog_manager.cc:1347] Generated new cluster ID: cfef60d95b4248c992e0537cf5eaae4c
I20260924 20:24:12.675575 7267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:12.691285 7267 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:12.691682 7267 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:12.699677 7267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba658970d25b49858b5faf1acc27a312: Generated new TSK 0
I20260924 20:24:12.700601 7267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:12.706930 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7175
2026-09-24T20:24:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:12 dist-test-slave-m1qq krb5kdc[7274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:12 dist-test-slave-m1qq krb5kdc[7274](info): set up 2 sockets
Sep 24 20:24:12 dist-test-slave-m1qq krb5kdc[7274](info): commencing operation
krb5kdc: starting...
W20260924 20:24:13.768307 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:13 dist-test-slave-m1qq krb5kdc[7274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:13.894320 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:14.005350 7287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:14.005630 7287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:14.005672 7287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:14.009709 7287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:14.009768 7287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:14.009788 7287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:14.009805 7287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:14.009819 7287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:14.015118 7287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:14.016137 7287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:14.017125 7287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:14.025851 7295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:14.025863 7292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:14.026146 7287 server_base.cc:1034] running on GCE node
W20260924 20:24:14.025851 7293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:14.026806 7287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:14.027932 7287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:14.029111 7287 hybrid_clock.cc:648] HybridClock initialized: now 1790281454029096 us; error 24 us; skew 500 ppm
Sep 24 20:24:14 dist-test-slave-m1qq krb5kdc[7274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:14.032073 7287 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:14.032938 7287 webserver.cc:460] Webserver started at http://127.0.0.1:45297/ using document root <none> and password file <none>
I20260924 20:24:14.033447 7287 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:14.033506 7287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:14.033659 7287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:14.035681 7287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "0b48a7d7260e4655a21aea903fb14ba8"
format_stamp: "Formatted at 2026-09-24 20:24:14 on dist-test-slave-m1qq"
server_key: "b5364e8ab8aa3ae84b2dfec3aa015726"
server_key_iv: "80928aac86236252ab39b5f7c0926e45"
server_key_version: "encryptionkey@0"
I20260924 20:24:14.036181 7287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "0b48a7d7260e4655a21aea903fb14ba8"
format_stamp: "Formatted at 2026-09-24 20:24:14 on dist-test-slave-m1qq"
server_key: "b5364e8ab8aa3ae84b2dfec3aa015726"
server_key_iv: "80928aac86236252ab39b5f7c0926e45"
server_key_version: "encryptionkey@0"
I20260924 20:24:14.039719 7287 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:24:14.041991 7302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:14.043255 7287 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:14.043401 7287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "0b48a7d7260e4655a21aea903fb14ba8"
format_stamp: "Formatted at 2026-09-24 20:24:14 on dist-test-slave-m1qq"
server_key: "b5364e8ab8aa3ae84b2dfec3aa015726"
server_key_iv: "80928aac86236252ab39b5f7c0926e45"
server_key_version: "encryptionkey@0"
I20260924 20:24:14.043514 7287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:14.066012 7287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:14.069464 7287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:14.069676 7287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:14.077036 7287 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42167
I20260924 20:24:14.077061 7354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42167 every 8 connection(s)
I20260924 20:24:14.078008 7287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:14.080782 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7287
I20260924 20:24:14.080927 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:14.080964 7355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:14.081271 19121 external_mini_cluster.cc:1428] Setting key 9f1c64a0928010c26107d4e9802b7d0c
I20260924 20:24:14.087407 7355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8: Bootstrap starting.
I20260924 20:24:14.089536 7355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:14.090251 7355 log.cc:824] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:24:14 dist-test-slave-m1qq krb5kdc[7274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:24:14.092350 7355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8: No bootstrap required, opened a new log
I20260924 20:24:14.095606 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:14.082723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51793 (local address 127.0.0.1:42167)
0924 20:24:14.083385 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:14.083400 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:14.083434 (+ 34us) server_negotiation.cc:407] Connection header received
0924 20:24:14.084136 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:14.084153 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:14.084710 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:14.085054 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:14.085871 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:14.086831 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:14.087798 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:14.088278 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:14.088448 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:14.091056 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:14.091063 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:14.091080 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:14.091121 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:14.092858 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:14.093308 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:14.093313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:14.093318 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:14.093424 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:14.093713 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:14.093717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:14.093719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:14.094096 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:14.094318 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:14.094555 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 20:24:14.094844 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":152,"threads_started":1}
I20260924 20:24:14.096021 7355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } }
I20260924 20:24:14.096259 7355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:14.096362 7355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b48a7d7260e4655a21aea903fb14ba8, State: Initialized, Role: FOLLOWER
I20260924 20:24:14.096838 7355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } }
I20260924 20:24:14.097000 7355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:14.097062 7355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:14.097155 7355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:14.098313 7355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } }
I20260924 20:24:14.098671 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:14.098666 7355 leader_election.cc:302] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b48a7d7260e4655a21aea903fb14ba8; no voters:
I20260924 20:24:14.098971 7355 leader_election.cc:288] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:14.099117 7360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:14.099356 7360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [term 1 LEADER]: Becoming Leader. State: Replica: 0b48a7d7260e4655a21aea903fb14ba8, State: Running, Role: LEADER
I20260924 20:24:14.099757 7360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } }
I20260924 20:24:14.100361 7355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:14.101266 7361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b48a7d7260e4655a21aea903fb14ba8. Latest consensus state: current_term: 1 leader_uuid: "0b48a7d7260e4655a21aea903fb14ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } } }
I20260924 20:24:14.101449 7361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:14.101688 7362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b48a7d7260e4655a21aea903fb14ba8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b48a7d7260e4655a21aea903fb14ba8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42167 } } }
I20260924 20:24:14.101791 7362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:14.102124 7374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:14.105350 7374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:14.110543 7358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:14.101986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33885 (local address 127.0.0.1:42167)
0924 20:24:14.102107 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:14.102113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:14.102128 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:24:14.102175 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:14.102178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:14.102229 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:14.102312 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:14.103291 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:14.104217 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:14.105036 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:14.105396 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:14.105499 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:14.107288 (+ 1789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:14.107292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:14.107294 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:14.107320 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:14.108762 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:14.109186 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:14.109191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:14.109193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:14.109271 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:14.109551 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:14.109555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:14.109557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:14.109765 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:14.109887 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:14.110239 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 20:24:14.110353 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:24:14.111572 7374 catalog_manager.cc:1347] Generated new cluster ID: fb25fc568f60497c8675c96127b4b397
I20260924 20:24:14.111634 7374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:14.136744 7374 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:14.138113 7374 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:14.148517 7374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b48a7d7260e4655a21aea903fb14ba8: Generated new TSK 0
I20260924 20:24:14.149401 7374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:14.163255 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7287
2026-09-24T20:24:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:14 dist-test-slave-m1qq krb5kdc[7386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:14 dist-test-slave-m1qq krb5kdc[7386](info): set up 2 sockets
Sep 24 20:24:14 dist-test-slave-m1qq krb5kdc[7386](info): commencing operation
krb5kdc: starting...
W20260924 20:24:15.244711 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:15 dist-test-slave-m1qq krb5kdc[7386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:15.382987 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:15.494741 7399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:15.495160 7399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:15.495265 7399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:15.499621 7399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:15.499738 7399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:15.499778 7399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:15.499819 7399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:15.499855 7399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:15.505591 7399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:15.506779 7399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:15.507876 7399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:15.516973 7405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:15.516999 7404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:15.517122 7407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:15.517254 7399 server_base.cc:1034] running on GCE node
I20260924 20:24:15.517839 7399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:15.519170 7399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:15.520364 7399 hybrid_clock.cc:648] HybridClock initialized: now 1790281455520331 us; error 47 us; skew 500 ppm
Sep 24 20:24:15 dist-test-slave-m1qq krb5kdc[7386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:15.523236 7399 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:15.524214 7399 webserver.cc:460] Webserver started at http://127.0.0.1:40355/ using document root <none> and password file <none>
I20260924 20:24:15.524807 7399 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:15.524904 7399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:15.525162 7399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:15.527408 7399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "e9a635199cd5409bad9e30e41b87c22e"
format_stamp: "Formatted at 2026-09-24 20:24:15 on dist-test-slave-m1qq"
server_key: "045962702933f1e1d409b57f0c0af580"
server_key_iv: "e8a02e8fe062047561f9ddf167c4fb9e"
server_key_version: "encryptionkey@0"
I20260924 20:24:15.528010 7399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "e9a635199cd5409bad9e30e41b87c22e"
format_stamp: "Formatted at 2026-09-24 20:24:15 on dist-test-slave-m1qq"
server_key: "045962702933f1e1d409b57f0c0af580"
server_key_iv: "e8a02e8fe062047561f9ddf167c4fb9e"
server_key_version: "encryptionkey@0"
I20260924 20:24:15.531997 7399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:24:15.534461 7414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:15.535821 7399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:15.535971 7399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "e9a635199cd5409bad9e30e41b87c22e"
format_stamp: "Formatted at 2026-09-24 20:24:15 on dist-test-slave-m1qq"
server_key: "045962702933f1e1d409b57f0c0af580"
server_key_iv: "e8a02e8fe062047561f9ddf167c4fb9e"
server_key_version: "encryptionkey@0"
I20260924 20:24:15.536089 7399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:15.549405 7399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:15.552542 7399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:15.552749 7399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:15.560078 7399 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33049
I20260924 20:24:15.560093 7466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33049 every 8 connection(s)
I20260924 20:24:15.561055 7399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:15.563946 7467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:15.569563 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7399
I20260924 20:24:15.569621 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e: Bootstrap starting.
I20260924 20:24:15.569722 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:15.570003 19121 external_mini_cluster.cc:1428] Setting key 2e73485a0319dbcbfe239f552620dfaa
I20260924 20:24:15.572253 7467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:15.573133 7467 log.cc:824] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:15.575519 7467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e: No bootstrap required, opened a new log
I20260924 20:24:15.579927 7467 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260924 20:24:15.580228 7467 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:15.580289 7467 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a635199cd5409bad9e30e41b87c22e, State: Initialized, Role: FOLLOWER
I20260924 20:24:15.580606 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:15.571589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42701 (local address 127.0.0.1:33049)
0924 20:24:15.572023 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:15.572042 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:15.572096 (+ 54us) server_negotiation.cc:407] Connection header received
0924 20:24:15.572950 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:15.572974 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:15.573430 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:15.573758 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:15.574240 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:15.575681 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:15.576623 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:15.577350 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:15.577511 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:15.578463 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:15.578472 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:15.578493 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:15.579036 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:15.579117 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:15.579148 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:15.579237 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:15.579396 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 20:24:15.579609 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":112,"threads_started":1}
I20260924 20:24:15.580775 7467 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260924 20:24:15.580953 7467 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:15.581022 7467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:15.581136 7467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:15.582050 7467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260924 20:24:15.582389 7467 leader_election.cc:302] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9a635199cd5409bad9e30e41b87c22e; no voters:
I20260924 20:24:15.582738 7467 leader_election.cc:288] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:15.582820 7472 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:15.583088 7472 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [term 1 LEADER]: Becoming Leader. State: Replica: e9a635199cd5409bad9e30e41b87c22e, State: Running, Role: LEADER
I20260924 20:24:15.583499 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:15.583467 7472 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } }
I20260924 20:24:15.584046 7467 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:15.585073 7473 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9a635199cd5409bad9e30e41b87c22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } } }
I20260924 20:24:15.585207 7473 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:15.585412 7473 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9a635199cd5409bad9e30e41b87c22e. Latest consensus state: current_term: 1 leader_uuid: "e9a635199cd5409bad9e30e41b87c22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9a635199cd5409bad9e30e41b87c22e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33049 } } }
I20260924 20:24:15.585500 7473 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:15.586568 7485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:15.589730 7485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:15.591344 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:15.585883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47751 (local address 127.0.0.1:33049)
0924 20:24:15.585988 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:15.585995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:15.586337 (+ 342us) server_negotiation.cc:407] Connection header received
0924 20:24:15.586741 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:15.586744 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:15.586790 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:15.586874 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:15.587210 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:15.587940 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:15.589064 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:15.589374 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:15.589469 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:15.590616 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:15.590620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:15.590623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:15.590636 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:15.590663 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:15.590677 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:15.590738 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:15.591162 (+ 424us) server_negotiation.cc:299] Negotiation successful
0924 20:24:15.591218 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 20:24:15.596320 7485 catalog_manager.cc:1347] Generated new cluster ID: 738a7baf516f45e490d01209db1e79ea
I20260924 20:24:15.596390 7485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:15.607836 7485 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:15.608932 7485 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:15.615856 7485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9a635199cd5409bad9e30e41b87c22e: Generated new TSK 0
I20260924 20:24:15.616580 7485 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:15.625516 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7399
2026-09-24T20:24:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:15 dist-test-slave-m1qq krb5kdc[7498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:15 dist-test-slave-m1qq krb5kdc[7498](info): set up 2 sockets
Sep 24 20:24:15 dist-test-slave-m1qq krb5kdc[7498](info): commencing operation
krb5kdc: starting...
W20260924 20:24:16.704615 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:16 dist-test-slave-m1qq krb5kdc[7498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:16.835296 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:16.948941 7511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:16.949216 7511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:16.949258 7511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:16.953274 7511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:16.953332 7511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:16.953351 7511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:16.953368 7511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:16.953383 7511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:16.958541 7511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:16.959594 7511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:16.960563 7511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:16.968997 7519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:16.969138 7517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:16.969030 7516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:16.969192 7511 server_base.cc:1034] running on GCE node
I20260924 20:24:16.969882 7511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:16.971040 7511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:16.972245 7511 hybrid_clock.cc:648] HybridClock initialized: now 1790281456972219 us; error 42 us; skew 500 ppm
Sep 24 20:24:16 dist-test-slave-m1qq krb5kdc[7498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:16.978888 7511 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:16.979853 7511 webserver.cc:460] Webserver started at http://127.0.0.1:42475/ using document root <none> and password file <none>
I20260924 20:24:16.980412 7511 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:16.980506 7511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:16.980741 7511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:16.982806 7511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "8cd9604a1ced40c2a992f11359b40c6e"
format_stamp: "Formatted at 2026-09-24 20:24:16 on dist-test-slave-m1qq"
server_key: "dfb961b875b5451c9bcd8a54eea58582"
server_key_iv: "b69be4d13e8b9c1e9e18adb297470439"
server_key_version: "encryptionkey@0"
I20260924 20:24:16.983422 7511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "8cd9604a1ced40c2a992f11359b40c6e"
format_stamp: "Formatted at 2026-09-24 20:24:16 on dist-test-slave-m1qq"
server_key: "dfb961b875b5451c9bcd8a54eea58582"
server_key_iv: "b69be4d13e8b9c1e9e18adb297470439"
server_key_version: "encryptionkey@0"
I20260924 20:24:16.987129 7511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:24:16.989529 7526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:16.990801 7511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:24:16.990943 7511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "8cd9604a1ced40c2a992f11359b40c6e"
format_stamp: "Formatted at 2026-09-24 20:24:16 on dist-test-slave-m1qq"
server_key: "dfb961b875b5451c9bcd8a54eea58582"
server_key_iv: "b69be4d13e8b9c1e9e18adb297470439"
server_key_version: "encryptionkey@0"
I20260924 20:24:16.991083 7511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:16.995719 7511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:16.998569 7511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:16.998723 7511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:17.006239 7511 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39167
I20260924 20:24:17.006265 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39167 every 8 connection(s)
I20260924 20:24:17.007253 7511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:17.010113 7579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:17.011679 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7511
I20260924 20:24:17.011821 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:17.012115 19121 external_mini_cluster.cc:1428] Setting key f5934b925f9f6f36b1e7a07ec48fafa8
I20260924 20:24:17.016415 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e: Bootstrap starting.
I20260924 20:24:17.017318 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:17.013497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37539 (local address 127.0.0.1:39167)
0924 20:24:17.013919 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:17.013934 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:17.014008 (+ 74us) server_negotiation.cc:407] Connection header received
0924 20:24:17.014664 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:17.014681 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:17.014950 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:17.015351 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:17.016144 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:17.016149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:17.016166 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:17.016176 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:17.016229 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:17.016247 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:17.016399 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:17.016465 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 20:24:17.016631 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":128,"threads_started":1}
I20260924 20:24:17.018793 7579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:17.019677 7579 log.cc:824] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:17.020229 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:17.022049 7579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e: No bootstrap required, opened a new log
I20260924 20:24:17.024111 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:17.022526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50041 (local address 127.0.0.1:39167)
0924 20:24:17.022655 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:17.022659 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:17.022832 (+ 173us) server_negotiation.cc:407] Connection header received
0924 20:24:17.022946 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:17.022948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:17.022975 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:17.023104 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:17.023740 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:17.023743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:17.023745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:17.023754 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:17.023776 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:17.023786 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:17.023851 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:17.023952 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 20:24:17.023981 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:24:17.026247 7579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260924 20:24:17.026506 7579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:17.026566 7579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd9604a1ced40c2a992f11359b40c6e, State: Initialized, Role: FOLLOWER
I20260924 20:24:17.026974 7579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260924 20:24:17.027177 7579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:17.027242 7579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:17.027360 7579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:17.028267 7579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260924 20:24:17.028613 7579 leader_election.cc:302] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8cd9604a1ced40c2a992f11359b40c6e; no voters:
I20260924 20:24:17.028911 7579 leader_election.cc:288] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:17.029023 7589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:17.029268 7589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [term 1 LEADER]: Becoming Leader. State: Replica: 8cd9604a1ced40c2a992f11359b40c6e, State: Running, Role: LEADER
I20260924 20:24:17.029593 7589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } }
I20260924 20:24:17.030120 7579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:17.031203 7590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8cd9604a1ced40c2a992f11359b40c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260924 20:24:17.031328 7590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:17.031702 7598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:17.031664 7591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cd9604a1ced40c2a992f11359b40c6e. Latest consensus state: current_term: 1 leader_uuid: "8cd9604a1ced40c2a992f11359b40c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8cd9604a1ced40c2a992f11359b40c6e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39167 } } }
I20260924 20:24:17.031822 7591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:17.034943 7598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:17.040665 7598 catalog_manager.cc:1347] Generated new cluster ID: b8f1cc418ccd44afbd16985109a935a6
I20260924 20:24:17.040736 7598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:17.059113 7598 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:17.059496 7598 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:17.063264 7598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8cd9604a1ced40c2a992f11359b40c6e: Generated new TSK 0
I20260924 20:24:17.063928 7598 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:17.071605 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7511
2026-09-24T20:24:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1445 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:17 dist-test-slave-m1qq krb5kdc[7610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:17 dist-test-slave-m1qq krb5kdc[7610](info): set up 2 sockets
Sep 24 20:24:17 dist-test-slave-m1qq krb5kdc[7610](info): commencing operation
krb5kdc: starting...
W20260924 20:24:18.108668 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:18.235127 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:18.345093 7623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:18.345405 7623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:18.345497 7623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:18.349524 7623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:18.349620 7623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:18.349659 7623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:18.349713 7623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:18.349746 7623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:18.355095 7623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:18.356277 7623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:18.357300 7623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:18.366611 7631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:18.366639 7629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:18.366740 7628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:18.367136 7623 server_base.cc:1034] running on GCE node
I20260924 20:24:18.367717 7623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:18.368948 7623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:18.370177 7623 hybrid_clock.cc:648] HybridClock initialized: now 1790281458370159 us; error 27 us; skew 500 ppm
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281458, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:18.373246 7623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:18.374171 7623 webserver.cc:460] Webserver started at http://127.0.0.1:44501/ using document root <none> and password file <none>
I20260924 20:24:18.374737 7623 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:18.374825 7623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:18.375109 7623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:18.377190 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "308c57a343f24dd0add0e6be6d7a0523"
format_stamp: "Formatted at 2026-09-24 20:24:18 on dist-test-slave-m1qq"
server_key: "839223049122356350a78abb36dc99d4"
server_key_iv: "fd37ec8d0984166a946302a5cb185f91"
server_key_version: "encryptionkey@0"
I20260924 20:24:18.377749 7623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "308c57a343f24dd0add0e6be6d7a0523"
format_stamp: "Formatted at 2026-09-24 20:24:18 on dist-test-slave-m1qq"
server_key: "839223049122356350a78abb36dc99d4"
server_key_iv: "fd37ec8d0984166a946302a5cb185f91"
server_key_version: "encryptionkey@0"
I20260924 20:24:18.381726 7623 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:24:18.384620 7638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:18.386022 7623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:24:18.386247 7623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "308c57a343f24dd0add0e6be6d7a0523"
format_stamp: "Formatted at 2026-09-24 20:24:18 on dist-test-slave-m1qq"
server_key: "839223049122356350a78abb36dc99d4"
server_key_iv: "fd37ec8d0984166a946302a5cb185f91"
server_key_version: "encryptionkey@0"
I20260924 20:24:18.386363 7623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:18.400000 7623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:18.402985 7623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:18.403267 7623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:18.410579 7623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32989
I20260924 20:24:18.410607 7690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32989 every 8 connection(s)
I20260924 20:24:18.411579 7623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:18.411765 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7623
I20260924 20:24:18.411921 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:18.412281 19121 external_mini_cluster.cc:1428] Setting key a9b8092ebb081f497a8da0911cf6b3fe
I20260924 20:24:18.414924 7691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:18.421429 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523: Bootstrap starting.
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:24:18.424124 7691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:18.424995 7691 log.cc:824] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:18.426909 7691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523: No bootstrap required, opened a new log
I20260924 20:24:18.427630 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:18.413820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:44629 (local address 127.0.0.1:32989)
0924 20:24:18.414267 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:18.414282 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:18.414464 (+ 182us) server_negotiation.cc:407] Connection header received
0924 20:24:18.415292 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:18.415310 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:18.415685 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:18.415955 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:18.416846 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:18.417847 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:18.418996 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:18.419562 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:18.419677 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:18.422600 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:18.422608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:18.422627 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:18.422664 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:18.424702 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:18.425199 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:18.425204 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:18.425208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:18.425318 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:18.425664 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:18.425667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:18.425669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:18.426052 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:18.426272 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:18.426588 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 20:24:18.426809 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":149,"threads_started":1}
I20260924 20:24:18.430465 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:18.430708 7691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260924 20:24:18.430943 7691 raft_consensus.cc:374] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:18.431035 7691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 308c57a343f24dd0add0e6be6d7a0523, State: Initialized, Role: FOLLOWER
I20260924 20:24:18.431494 7691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260924 20:24:18.431650 7691 raft_consensus.cc:388] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:18.431722 7691 raft_consensus.cc:482] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:18.431826 7691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:18.433070 7691 raft_consensus.cc:504] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260924 20:24:18.433451 7691 leader_election.cc:302] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 308c57a343f24dd0add0e6be6d7a0523; no voters:
I20260924 20:24:18.433751 7691 leader_election.cc:288] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:18.433872 7701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:18.434198 7701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [term 1 LEADER]: Becoming Leader. State: Replica: 308c57a343f24dd0add0e6be6d7a0523, State: Running, Role: LEADER
I20260924 20:24:18.434556 7701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } }
I20260924 20:24:18.435070 7691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:18.436357 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "308c57a343f24dd0add0e6be6d7a0523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260924 20:24:18.436478 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:18.436754 7702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 308c57a343f24dd0add0e6be6d7a0523. Latest consensus state: current_term: 1 leader_uuid: "308c57a343f24dd0add0e6be6d7a0523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "308c57a343f24dd0add0e6be6d7a0523" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32989 } } }
I20260924 20:24:18.436914 7702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:18.437431 7711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:18.441011 7711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:18.442585 7694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:18.432699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:47371 (local address 127.0.0.1:32989)
0924 20:24:18.432836 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:18.432840 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:18.432853 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:24:18.432933 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:18.432937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:18.432977 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:18.433076 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:18.434124 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:18.434879 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:18.436099 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:18.436483 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:18.436571 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:18.438499 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:18.438502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:18.438506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:18.438531 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:18.440164 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:18.440709 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:18.440713 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:18.440715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:18.440798 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:18.441191 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:18.441195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:18.441197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:18.441414 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:18.441558 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:18.442335 (+ 777us) server_negotiation.cc:299] Negotiation successful
0924 20:24:18.442433 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 20:24:18.446556 7711 catalog_manager.cc:1347] Generated new cluster ID: 66c71f0ee2c24bf08e46dff1abd48c81
I20260924 20:24:18.446641 7711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:18.474671 7711 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:18.475715 7711 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:18.480753 7711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 308c57a343f24dd0add0e6be6d7a0523: Generated new TSK 0
I20260924 20:24:18.481544 7711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:18.487555 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7623
W20260924 20:24:18.493197 7608 connection.cc:535] client connection to 127.0.0.1:32989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32989): Connection reset by peer (error 104)
2026-09-24T20:24:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7722](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7722](info): set up 2 sockets
Sep 24 20:24:18 dist-test-slave-m1qq krb5kdc[7722](info): commencing operation
krb5kdc: starting...
W20260924 20:24:19.552434 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:19 dist-test-slave-m1qq krb5kdc[7722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:19.678107 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:19.788215 7735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:19.788468 7735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:19.788511 7735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:19.792383 7735 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:19.792434 7735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:19.792454 7735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:19.792469 7735 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:19.792484 7735 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:19.797711 7735 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:19.798666 7735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:19.799690 7735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:19.808259 7743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:19.808260 7740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:19.808262 7741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:19.808494 7735 server_base.cc:1034] running on GCE node
I20260924 20:24:19.809120 7735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:19.810350 7735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:19.811532 7735 hybrid_clock.cc:648] HybridClock initialized: now 1790281459811503 us; error 58 us; skew 500 ppm
Sep 24 20:24:19 dist-test-slave-m1qq krb5kdc[7722](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281459, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:19.814359 7735 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:19.815443 7735 webserver.cc:460] Webserver started at http://127.0.0.1:34975/ using document root <none> and password file <none>
I20260924 20:24:19.816012 7735 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:19.816107 7735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:19.816341 7735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:19.818408 7735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "bc5f5c01d1404e2584e80c760058bb23"
format_stamp: "Formatted at 2026-09-24 20:24:19 on dist-test-slave-m1qq"
server_key: "06d6e40d82c8e1541ec330999e6c569d"
server_key_iv: "65271d39494c9cde799afc2459434f80"
server_key_version: "encryptionkey@0"
I20260924 20:24:19.818971 7735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "bc5f5c01d1404e2584e80c760058bb23"
format_stamp: "Formatted at 2026-09-24 20:24:19 on dist-test-slave-m1qq"
server_key: "06d6e40d82c8e1541ec330999e6c569d"
server_key_iv: "65271d39494c9cde799afc2459434f80"
server_key_version: "encryptionkey@0"
I20260924 20:24:19.822697 7735 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:19.825238 7750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:19.826429 7735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:24:19.826570 7735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "bc5f5c01d1404e2584e80c760058bb23"
format_stamp: "Formatted at 2026-09-24 20:24:19 on dist-test-slave-m1qq"
server_key: "06d6e40d82c8e1541ec330999e6c569d"
server_key_iv: "65271d39494c9cde799afc2459434f80"
server_key_version: "encryptionkey@0"
I20260924 20:24:19.826683 7735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:19.837513 7735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:19.840579 7735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:19.840791 7735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:19.847908 7735 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33941
I20260924 20:24:19.847924 7802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33941 every 8 connection(s)
I20260924 20:24:19.848814 7735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:19.851428 7803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:19.854555 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7735
I20260924 20:24:19.854717 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:19.855048 19121 external_mini_cluster.cc:1428] Setting key 2cfcce27a8e2cb7e34e91ab3b4467cb7
I20260924 20:24:19.857623 7803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23: Bootstrap starting.
I20260924 20:24:19.859827 7803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:19.860543 7803 log.cc:824] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:19.862450 7803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23: No bootstrap required, opened a new log
I20260924 20:24:19.863885 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:19.856482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:37473 (local address 127.0.0.1:33941)
0924 20:24:19.856903 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:19.856917 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:19.856952 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:24:19.857568 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:19.857585 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:19.858001 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:19.858328 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:19.858731 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:19.859761 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:19.860693 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:19.861150 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:19.861319 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:19.862210 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:19.862216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:19.862238 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:19.862649 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:19.862702 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:19.862724 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:19.862857 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:19.863055 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 20:24:19.863199 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":123,"threads_started":1}
I20260924 20:24:19.866320 7803 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260924 20:24:19.866602 7803 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:19.866667 7803 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc5f5c01d1404e2584e80c760058bb23, State: Initialized, Role: FOLLOWER
I20260924 20:24:19.866916 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:19.867163 7803 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260924 20:24:19.867306 7803 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:19.867364 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:19.867470 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:19.868646 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260924 20:24:19.868975 7803 leader_election.cc:302] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc5f5c01d1404e2584e80c760058bb23; no voters:
I20260924 20:24:19.869303 7803 leader_election.cc:288] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:19.869655 7813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:19.869936 7813 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [term 1 LEADER]: Becoming Leader. State: Replica: bc5f5c01d1404e2584e80c760058bb23, State: Running, Role: LEADER
I20260924 20:24:19.870296 7813 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } }
I20260924 20:24:19.870566 7803 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:19.871490 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc5f5c01d1404e2584e80c760058bb23. Latest consensus state: current_term: 1 leader_uuid: "bc5f5c01d1404e2584e80c760058bb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260924 20:24:19.871598 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:19.871891 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc5f5c01d1404e2584e80c760058bb23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc5f5c01d1404e2584e80c760058bb23" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33941 } } }
I20260924 20:24:19.871986 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:19.872329 7822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:19.873996 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:19.869151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:56997 (local address 127.0.0.1:33941)
0924 20:24:19.869274 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:19.869279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:19.869444 (+ 165us) server_negotiation.cc:407] Connection header received
0924 20:24:19.869551 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:19.869555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:19.869603 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:19.869714 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:19.870139 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:19.870898 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:19.872035 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:19.872400 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:19.872501 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:19.873524 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:19.873527 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:19.873530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:19.873542 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:19.873574 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:19.873584 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:19.873687 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:19.873818 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 20:24:19.873850 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:24:19.876108 7822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:19.881440 7822 catalog_manager.cc:1347] Generated new cluster ID: 50e07d9a57ad4a50914eeccad71235c6
I20260924 20:24:19.881510 7822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:19.898339 7822 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:19.899619 7822 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:19.912664 7822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc5f5c01d1404e2584e80c760058bb23: Generated new TSK 0
I20260924 20:24:19.913643 7822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:19.923074 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7735
W20260924 20:24:19.928442 7720 connection.cc:535] client connection to 127.0.0.1:33941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33941): Connection reset by peer (error 104)
2026-09-24T20:24:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:19 dist-test-slave-m1qq krb5kdc[7834](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:19 dist-test-slave-m1qq krb5kdc[7834](info): set up 2 sockets
Sep 24 20:24:19 dist-test-slave-m1qq krb5kdc[7834](info): commencing operation
krb5kdc: starting...
W20260924 20:24:20.960911 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:21 dist-test-slave-m1qq krb5kdc[7834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:21.090005 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37913
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:21.200102 7847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:21.200371 7847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:21.200412 7847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:21.204450 7847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:21.204516 7847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:21.204535 7847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:21.204552 7847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:21.204568 7847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:21.209874 7847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:21.211051 7847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:21.212117 7847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:21.221479 7853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:21.221588 7847 server_base.cc:1034] running on GCE node
W20260924 20:24:21.221479 7855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:21.221552 7852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:21.222419 7847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:21.223704 7847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:21.224882 7847 hybrid_clock.cc:648] HybridClock initialized: now 1790281461224853 us; error 37 us; skew 500 ppm
Sep 24 20:24:21 dist-test-slave-m1qq krb5kdc[7834](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:21.228080 7847 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:21.229116 7847 webserver.cc:460] Webserver started at http://127.0.0.1:36003/ using document root <none> and password file <none>
I20260924 20:24:21.229714 7847 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:21.229811 7847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:21.230075 7847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:21.232426 7847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "aa15917f7d704bd69beff932a02cd331"
format_stamp: "Formatted at 2026-09-24 20:24:21 on dist-test-slave-m1qq"
server_key: "1290b5dbf12f15643ac752e5e052eebc"
server_key_iv: "0b749a5191c8f2c86ecf6509f86249ef"
server_key_version: "encryptionkey@0"
I20260924 20:24:21.233049 7847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "aa15917f7d704bd69beff932a02cd331"
format_stamp: "Formatted at 2026-09-24 20:24:21 on dist-test-slave-m1qq"
server_key: "1290b5dbf12f15643ac752e5e052eebc"
server_key_iv: "0b749a5191c8f2c86ecf6509f86249ef"
server_key_version: "encryptionkey@0"
I20260924 20:24:21.237376 7847 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260924 20:24:21.240309 7862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:21.241811 7847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:21.241957 7847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "aa15917f7d704bd69beff932a02cd331"
format_stamp: "Formatted at 2026-09-24 20:24:21 on dist-test-slave-m1qq"
server_key: "1290b5dbf12f15643ac752e5e052eebc"
server_key_iv: "0b749a5191c8f2c86ecf6509f86249ef"
server_key_version: "encryptionkey@0"
I20260924 20:24:21.242093 7847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:21.247092 7847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:21.250794 7847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:21.251133 7847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:21.258893 7847 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37913
I20260924 20:24:21.258949 7914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37913 every 8 connection(s)
I20260924 20:24:21.259964 7847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:21.262818 7915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:21.266423 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7847
I20260924 20:24:21.266587 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:21.266933 19121 external_mini_cluster.cc:1428] Setting key 38ba9ff1db053f4e10ed78cfca78c496
I20260924 20:24:21.268792 7915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331: Bootstrap starting.
I20260924 20:24:21.271584 7915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:21.272501 7915 log.cc:824] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:21.273049 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:21.268630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:59637 (local address 127.0.0.1:37913)
0924 20:24:21.269112 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:21.269130 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:21.269182 (+ 52us) server_negotiation.cc:407] Connection header received
0924 20:24:21.270066 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:21.270088 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:21.270462 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:21.270813 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:21.271605 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:21.271611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:21.271628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:21.271644 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:21.271717 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:21.271749 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:21.271865 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:21.271997 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 20:24:21.272159 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":121,"threads_started":1}
I20260924 20:24:21.274703 7915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331: No bootstrap required, opened a new log
I20260924 20:24:21.275975 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:21.279053 7915 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } }
I20260924 20:24:21.279364 7915 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:21.279435 7915 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa15917f7d704bd69beff932a02cd331, State: Initialized, Role: FOLLOWER
I20260924 20:24:21.279911 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:21.277971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:53629 (local address 127.0.0.1:37913)
0924 20:24:21.278102 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:21.278106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:21.278362 (+ 256us) server_negotiation.cc:407] Connection header received
0924 20:24:21.278480 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:21.278482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:21.278511 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:21.278574 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:21.279446 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:21.279450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:21.279453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:21.279467 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:21.279520 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:21.279533 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:21.279618 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:21.279760 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 20:24:21.279803 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:24:21.279966 7915 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } }
I20260924 20:24:21.280126 7915 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:21.280208 7915 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:21.280320 7915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:21.281404 7915 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } }
I20260924 20:24:21.281706 7915 leader_election.cc:302] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa15917f7d704bd69beff932a02cd331; no voters:
I20260924 20:24:21.281949 7915 leader_election.cc:288] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:21.282110 7925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:21.282423 7925 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [term 1 LEADER]: Becoming Leader. State: Replica: aa15917f7d704bd69beff932a02cd331, State: Running, Role: LEADER
I20260924 20:24:21.282780 7925 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } }
I20260924 20:24:21.283316 7915 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:21.284571 7927 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa15917f7d704bd69beff932a02cd331. Latest consensus state: current_term: 1 leader_uuid: "aa15917f7d704bd69beff932a02cd331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } } }
I20260924 20:24:21.284682 7927 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:21.285099 7934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:21.286490 7926 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa15917f7d704bd69beff932a02cd331" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa15917f7d704bd69beff932a02cd331" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37913 } } }
I20260924 20:24:21.286588 7926 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:21.288605 7934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:21.294518 7934 catalog_manager.cc:1347] Generated new cluster ID: a6fce680ef4145c4976361ce625dcf41
I20260924 20:24:21.294603 7934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:21.307078 7934 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:21.307467 7934 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:21.320667 7934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa15917f7d704bd69beff932a02cd331: Generated new TSK 0
I20260924 20:24:21.321450 7934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:21.327292 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7847
2026-09-24T20:24:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:21 dist-test-slave-m1qq krb5kdc[7946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:21 dist-test-slave-m1qq krb5kdc[7946](info): set up 2 sockets
Sep 24 20:24:21 dist-test-slave-m1qq krb5kdc[7946](info): commencing operation
krb5kdc: starting...
W20260924 20:24:22.400372 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:22.530366 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45249
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:22.643927 7959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:22.644233 7959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:22.644279 7959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:22.648519 7959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:22.648586 7959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:22.648607 7959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:22.648623 7959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:22.648639 7959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:22.653925 7959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45249
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:22.654961 7959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:22.656009 7959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:22.664821 7964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:22.664870 7965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:22.664848 7967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:22.665086 7959 server_base.cc:1034] running on GCE node
I20260924 20:24:22.665692 7959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:22.666785 7959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:22.667949 7959 hybrid_clock.cc:648] HybridClock initialized: now 1790281462667952 us; error 53 us; skew 500 ppm
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[7946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281462, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:22.674569 7959 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:22.675684 7959 webserver.cc:460] Webserver started at http://127.0.0.1:34385/ using document root <none> and password file <none>
I20260924 20:24:22.676285 7959 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:22.676348 7959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:22.676566 7959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:22.678745 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "cbf9c676842444998430be5e9694bab3"
format_stamp: "Formatted at 2026-09-24 20:24:22 on dist-test-slave-m1qq"
server_key: "fcb4b53a983638d0a50a848b2a14d744"
server_key_iv: "798e58ad1c04f287de00c2ddc7bcbf1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:22.679378 7959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "cbf9c676842444998430be5e9694bab3"
format_stamp: "Formatted at 2026-09-24 20:24:22 on dist-test-slave-m1qq"
server_key: "fcb4b53a983638d0a50a848b2a14d744"
server_key_iv: "798e58ad1c04f287de00c2ddc7bcbf1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:22.683379 7959 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:24:22.686022 7974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:22.687573 7959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:22.687726 7959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "cbf9c676842444998430be5e9694bab3"
format_stamp: "Formatted at 2026-09-24 20:24:22 on dist-test-slave-m1qq"
server_key: "fcb4b53a983638d0a50a848b2a14d744"
server_key_iv: "798e58ad1c04f287de00c2ddc7bcbf1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:22.687846 7959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:22.705331 7959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:22.708693 7959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:22.708930 7959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:22.716631 7959 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45249
I20260924 20:24:22.716759 8026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45249 every 8 connection(s)
I20260924 20:24:22.717686 7959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:22.720701 8027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:22.726769 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3: Bootstrap starting.
I20260924 20:24:22.727171 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 7959
I20260924 20:24:22.727324 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:22.727643 19121 external_mini_cluster.cc:1428] Setting key d69e9f10b21c12fa8f20aea1003efd6e
I20260924 20:24:22.729987 8027 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:22.730823 8027 log.cc:824] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:22.733362 8027 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3: No bootstrap required, opened a new log
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[7946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281462, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 20:24:22.738566 8027 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260924 20:24:22.738867 8027 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:22.738953 8027 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbf9c676842444998430be5e9694bab3, State: Initialized, Role: FOLLOWER
I20260924 20:24:22.739643 8027 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260924 20:24:22.739861 8027 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:22.739964 8027 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:22.740080 8027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:22.741093 8027 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260924 20:24:22.741473 8027 leader_election.cc:302] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbf9c676842444998430be5e9694bab3; no voters:
I20260924 20:24:22.741868 8027 leader_election.cc:288] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:22.741943 8032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:22.742218 8032 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [term 1 LEADER]: Becoming Leader. State: Replica: cbf9c676842444998430be5e9694bab3, State: Running, Role: LEADER
I20260924 20:24:22.742578 8032 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } }
I20260924 20:24:22.743422 8027 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:22.744531 8034 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbf9c676842444998430be5e9694bab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } } }
I20260924 20:24:22.744948 8034 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:22.744845 8033 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbf9c676842444998430be5e9694bab3. Latest consensus state: current_term: 1 leader_uuid: "cbf9c676842444998430be5e9694bab3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbf9c676842444998430be5e9694bab3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45249 } } }
I20260924 20:24:22.745380 8033 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:22.745754 8041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:22.745777 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:22.729265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:44329 (local address 127.0.0.1:45249)
0924 20:24:22.729775 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:22.729791 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:22.729829 (+ 38us) server_negotiation.cc:407] Connection header received
0924 20:24:22.730691 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:22.730712 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:22.731223 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:22.731628 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:22.732652 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:22.733767 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:22.735113 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:22.735676 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:22.735809 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:22.739289 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:22.739297 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:22.739320 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:22.739361 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:22.741486 (+ 2125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:22.742091 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:22.742096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:22.742101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:22.742355 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:22.742805 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:22.742807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:22.742812 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:22.743321 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:22.743563 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:22.743894 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 20:24:22.744148 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":137,"threads_started":1}
I20260924 20:24:22.749074 8041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:22.749430 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:22.756518 8041 catalog_manager.cc:1347] Generated new cluster ID: 9797168f6a7f4e519e7743f2857d2281
I20260924 20:24:22.756619 8041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:22.761902 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:22.752019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:46399 (local address 127.0.0.1:45249)
0924 20:24:22.752138 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:22.752143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:22.752369 (+ 226us) server_negotiation.cc:407] Connection header received
0924 20:24:22.752475 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:22.752479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:22.752534 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:22.752629 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:22.753452 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:22.754356 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:22.756616 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:22.756878 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:22.756937 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:22.758502 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:22.758506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:22.758509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:22.758541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:22.760083 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:22.760575 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:22.760579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:22.760581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:22.760662 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:22.760993 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:22.760996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:22.760998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:22.761175 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:22.761330 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:22.761612 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 20:24:22.761713 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:24:22.770825 8041 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:22.771927 8041 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:22.777806 8041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbf9c676842444998430be5e9694bab3: Generated new TSK 0
I20260924 20:24:22.778609 8041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:22.790934 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 7959
W20260924 20:24:22.796492 7944 connection.cc:535] client connection to 127.0.0.1:45249 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45249): Connection reset by peer (error 104)
2026-09-24T20:24:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1462 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[8058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[8058](info): set up 2 sockets
Sep 24 20:24:22 dist-test-slave-m1qq krb5kdc[8058](info): commencing operation
krb5kdc: starting...
W20260924 20:24:23.856930 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:23 dist-test-slave-m1qq krb5kdc[8058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:23.985756 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:24.096326 8071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:24.096650 8071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:24.096741 8071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:24.100975 8071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:24.101075 8071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:24.101115 8071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:24.101173 8071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:24.101221 8071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:24.106906 8071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:24.108173 8071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:24.109344 8071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:24.118369 8079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:24.118485 8071 server_base.cc:1034] running on GCE node
W20260924 20:24:24.118360 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:24.118484 8077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:24.119467 8071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:24.120654 8071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:24.121827 8071 hybrid_clock.cc:648] HybridClock initialized: now 1790281464121812 us; error 28 us; skew 500 ppm
Sep 24 20:24:24 dist-test-slave-m1qq krb5kdc[8058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:24.124724 8071 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:24.125708 8071 webserver.cc:460] Webserver started at http://127.0.0.1:45163/ using document root <none> and password file <none>
I20260924 20:24:24.126274 8071 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:24.126344 8071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:24.126574 8071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:24.128789 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "2d5d27aa4ec14e9c9f19488335022f26"
format_stamp: "Formatted at 2026-09-24 20:24:24 on dist-test-slave-m1qq"
server_key: "9f7e1b86dab2cb5b3c3128113296eb25"
server_key_iv: "d1b45bc3c389c74710179ea31dbba01d"
server_key_version: "encryptionkey@0"
I20260924 20:24:24.129386 8071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "2d5d27aa4ec14e9c9f19488335022f26"
format_stamp: "Formatted at 2026-09-24 20:24:24 on dist-test-slave-m1qq"
server_key: "9f7e1b86dab2cb5b3c3128113296eb25"
server_key_iv: "d1b45bc3c389c74710179ea31dbba01d"
server_key_version: "encryptionkey@0"
I20260924 20:24:24.133219 8071 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260924 20:24:24.135780 8086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:24.137012 8071 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:24:24.137148 8071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "2d5d27aa4ec14e9c9f19488335022f26"
format_stamp: "Formatted at 2026-09-24 20:24:24 on dist-test-slave-m1qq"
server_key: "9f7e1b86dab2cb5b3c3128113296eb25"
server_key_iv: "d1b45bc3c389c74710179ea31dbba01d"
server_key_version: "encryptionkey@0"
I20260924 20:24:24.137311 8071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:24.155285 8071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:24.158303 8071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:24.158516 8071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:24.165692 8071 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36341
I20260924 20:24:24.165712 8138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36341 every 8 connection(s)
I20260924 20:24:24.166677 8071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:24.169561 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:24.172322 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8071
I20260924 20:24:24.172487 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:24.172798 19121 external_mini_cluster.cc:1428] Setting key b55431acf098e171161b023b18bcc10f
I20260924 20:24:24.176012 8139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26: Bootstrap starting.
I20260924 20:24:24.178548 8139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:24.179443 8139 log.cc:824] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:24.181648 8139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26: No bootstrap required, opened a new log
I20260924 20:24:24.182861 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:24.174316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:56855 (local address 127.0.0.1:36341)
0924 20:24:24.174773 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:24.174791 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:24.174832 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:24:24.175596 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:24.175616 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:24.176014 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:24.176384 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:24.176794 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:24.177939 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:24.179237 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:24.179858 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:24.180186 (+ 328us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:24.180999 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:24.181004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:24.181018 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:24.181411 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:24.181469 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:24.181501 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:24.181614 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:24.181870 (+ 256us) server_negotiation.cc:299] Negotiation successful
0924 20:24:24.182025 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":160,"threads_started":1}
I20260924 20:24:24.185457 8139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } }
I20260924 20:24:24.185681 8139 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:24.185745 8139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d5d27aa4ec14e9c9f19488335022f26, State: Initialized, Role: FOLLOWER
I20260924 20:24:24.186021 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:24.186220 8139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } }
I20260924 20:24:24.186363 8139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:24.186429 8139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:24.186539 8139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:24.187618 8139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } }
I20260924 20:24:24.187955 8139 leader_election.cc:302] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d5d27aa4ec14e9c9f19488335022f26; no voters:
I20260924 20:24:24.188298 8139 leader_election.cc:288] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:24.188409 8148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:24.188635 8148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [term 1 LEADER]: Becoming Leader. State: Replica: 2d5d27aa4ec14e9c9f19488335022f26, State: Running, Role: LEADER
I20260924 20:24:24.188941 8148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } }
I20260924 20:24:24.189638 8139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:24.190616 8150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d5d27aa4ec14e9c9f19488335022f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } } }
I20260924 20:24:24.190744 8150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:24.191083 8151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d5d27aa4ec14e9c9f19488335022f26. Latest consensus state: current_term: 1 leader_uuid: "2d5d27aa4ec14e9c9f19488335022f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d5d27aa4ec14e9c9f19488335022f26" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36341 } } }
I20260924 20:24:24.191174 8151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:24.191442 8158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:24.193153 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:24.188305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:51031 (local address 127.0.0.1:36341)
0924 20:24:24.188420 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:24.188424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:24.188662 (+ 238us) server_negotiation.cc:407] Connection header received
0924 20:24:24.188865 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:24.188868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:24.188917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:24.188962 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:24.189356 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:24.190011 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:24.191289 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:24.191637 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:24.191746 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:24.192658 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:24.192662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:24.192665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:24.192678 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:24.192706 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:24.192719 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:24.192807 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:24.192994 (+ 187us) server_negotiation.cc:299] Negotiation successful
0924 20:24:24.193029 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:24:24.195538 8158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:24.201297 8158 catalog_manager.cc:1347] Generated new cluster ID: d8720d52a55645a5a5b4c4bc99e66690
I20260924 20:24:24.201367 8158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:24.262355 8158 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:24.263434 8158 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:24.268303 8158 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d5d27aa4ec14e9c9f19488335022f26: Generated new TSK 0
I20260924 20:24:24.269008 8158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:24.275900 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8071
W20260924 20:24:24.280761 8056 connection.cc:535] client connection to 127.0.0.1:36341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36341): Connection reset by peer (error 104)
2026-09-24T20:24:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1484 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:24 dist-test-slave-m1qq krb5kdc[8170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:24 dist-test-slave-m1qq krb5kdc[8170](info): set up 2 sockets
Sep 24 20:24:24 dist-test-slave-m1qq krb5kdc[8170](info): commencing operation
krb5kdc: starting...
W20260924 20:24:25.436544 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:25 dist-test-slave-m1qq krb5kdc[8170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 20:24:25.566708 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:25.681751 8183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:25.682086 8183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:25.682173 8183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:25.686255 8183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:25.686324 8183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:25.686344 8183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:25.686362 8183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:25.686376 8183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:25.692054 8183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:25.693190 8183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:25.694255 8183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:25.703461 8188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:25.703487 8191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:25.703438 8189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:25.703785 8183 server_base.cc:1034] running on GCE node
I20260924 20:24:25.704370 8183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:25.705552 8183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:25.706738 8183 hybrid_clock.cc:648] HybridClock initialized: now 1790281465706706 us; error 42 us; skew 500 ppm
Sep 24 20:24:25 dist-test-slave-m1qq krb5kdc[8170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:25.709718 8183 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 20:24:25.710676 8183 webserver.cc:460] Webserver started at http://127.0.0.1:41305/ using document root <none> and password file <none>
I20260924 20:24:25.711374 8183 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:25.711442 8183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:25.711616 8183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:25.713809 8183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "88f6ad8fff434a09967244e74f0a4ffa"
format_stamp: "Formatted at 2026-09-24 20:24:25 on dist-test-slave-m1qq"
server_key: "b21caeaf58298e58b78744045da32b4e"
server_key_iv: "11d2a189593b8c5c1cf2fee649caef61"
server_key_version: "encryptionkey@0"
I20260924 20:24:25.714378 8183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "88f6ad8fff434a09967244e74f0a4ffa"
format_stamp: "Formatted at 2026-09-24 20:24:25 on dist-test-slave-m1qq"
server_key: "b21caeaf58298e58b78744045da32b4e"
server_key_iv: "11d2a189593b8c5c1cf2fee649caef61"
server_key_version: "encryptionkey@0"
I20260924 20:24:25.718590 8183 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260924 20:24:25.721340 8198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:25.722731 8183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:25.722887 8183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "88f6ad8fff434a09967244e74f0a4ffa"
format_stamp: "Formatted at 2026-09-24 20:24:25 on dist-test-slave-m1qq"
server_key: "b21caeaf58298e58b78744045da32b4e"
server_key_iv: "11d2a189593b8c5c1cf2fee649caef61"
server_key_version: "encryptionkey@0"
I20260924 20:24:25.723011 8183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:25.727695 8183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:25.730706 8183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:25.730909 8183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:25.737914 8183 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43747
I20260924 20:24:25.737927 8250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43747 every 8 connection(s)
I20260924 20:24:25.738885 8183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:25.741736 8251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:25.743312 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8183
I20260924 20:24:25.743469 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:25.743750 19121 external_mini_cluster.cc:1428] Setting key 983684857203a4729dad6e2e77890164
I20260924 20:24:25.748632 8251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa: Bootstrap starting.
I20260924 20:24:25.749507 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:25.745203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:52681 (local address 127.0.0.1:43747)
0924 20:24:25.745634 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:25.745655 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:25.745701 (+ 46us) server_negotiation.cc:407] Connection header received
0924 20:24:25.746565 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:25.746590 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:25.746962 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:25.747369 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:25.748145 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:25.748150 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:25.748165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:25.748176 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:25.748247 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:25.748273 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:25.748412 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:25.748478 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 20:24:25.748631 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":113,"threads_started":1}
I20260924 20:24:25.751163 8251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:25.751988 8251 log.cc:824] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:25.752378 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:25.754261 8251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa: No bootstrap required, opened a new log
I20260924 20:24:25.756336 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:25.754638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.200:33539 (local address 127.0.0.1:43747)
0924 20:24:25.754762 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:25.754766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:25.754923 (+ 157us) server_negotiation.cc:407] Connection header received
0924 20:24:25.755091 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:25.755093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:25.755122 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:25.755182 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:25.755881 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:25.755883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:25.755885 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:25.755895 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:25.755919 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:25.755929 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:25.755995 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:25.756090 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 20:24:25.756122 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 20:24:25.758656 8251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } }
I20260924 20:24:25.758958 8251 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:25.759037 8251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f6ad8fff434a09967244e74f0a4ffa, State: Initialized, Role: FOLLOWER
I20260924 20:24:25.759444 8251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } }
I20260924 20:24:25.759593 8251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:25.759681 8251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:25.759783 8251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:25.760674 8251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } }
I20260924 20:24:25.761023 8251 leader_election.cc:302] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f6ad8fff434a09967244e74f0a4ffa; no voters:
I20260924 20:24:25.761325 8251 leader_election.cc:288] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:25.761445 8261 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:25.761690 8261 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [term 1 LEADER]: Becoming Leader. State: Replica: 88f6ad8fff434a09967244e74f0a4ffa, State: Running, Role: LEADER
I20260924 20:24:25.762040 8261 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } }
I20260924 20:24:25.762514 8251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:25.763890 8262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f6ad8fff434a09967244e74f0a4ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } } }
I20260924 20:24:25.763880 8263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f6ad8fff434a09967244e74f0a4ffa. Latest consensus state: current_term: 1 leader_uuid: "88f6ad8fff434a09967244e74f0a4ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f6ad8fff434a09967244e74f0a4ffa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43747 } } }
I20260924 20:24:25.764022 8263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:25.764022 8262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:25.765236 8271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:25.768028 8271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:25.773185 8271 catalog_manager.cc:1347] Generated new cluster ID: 24d542a222a94565b2c6fdcb485d6f43
I20260924 20:24:25.773263 8271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:25.789803 8271 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:25.790124 8271 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:25.795971 8271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f6ad8fff434a09967244e74f0a4ffa: Generated new TSK 0
I20260924 20:24:25.796617 8271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:25.800570 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8183
2026-09-24T20:24:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:25 dist-test-slave-m1qq krb5kdc[8282](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:25 dist-test-slave-m1qq krb5kdc[8282](info): set up 2 sockets
Sep 24 20:24:25 dist-test-slave-m1qq krb5kdc[8282](info): commencing operation
krb5kdc: starting...
W20260924 20:24:26.845039 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:26 dist-test-slave-m1qq krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:26.974844 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43681
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:43273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43681
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:27.089536 8295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:27.089823 8295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:27.089865 8295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:27.093952 8295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:27.094017 8295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:27.094038 8295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:27.094055 8295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:27.094071 8295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:27.099493 8295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:43273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43681
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43681
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:27.100656 8295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:27.101624 8295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:27.111258 8303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:27.111263 8301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:27.111445 8300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:27.111768 8295 server_base.cc:1034] running on GCE node
I20260924 20:24:27.112382 8295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:27.113677 8295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:27.114970 8295 hybrid_clock.cc:648] HybridClock initialized: now 1790281467114878 us; error 101 us; skew 500 ppm
Sep 24 20:24:27 dist-test-slave-m1qq krb5kdc[8282](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281467, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:27.118191 8295 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:27.119211 8295 webserver.cc:460] Webserver started at http://127.18.172.126:45877/ using document root <none> and password file <none>
I20260924 20:24:27.119839 8295 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:27.119954 8295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:27.120198 8295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:27.122292 8295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "fec25743e03f4c07badf221e662f6dbd"
format_stamp: "Formatted at 2026-09-24 20:24:27 on dist-test-slave-m1qq"
server_key: "46963523878af86bdce72cb6f79bc3bd"
server_key_iv: "99ae639fce7b5a9901b52bbd8fef800d"
server_key_version: "encryptionkey@0"
I20260924 20:24:27.122887 8295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "fec25743e03f4c07badf221e662f6dbd"
format_stamp: "Formatted at 2026-09-24 20:24:27 on dist-test-slave-m1qq"
server_key: "46963523878af86bdce72cb6f79bc3bd"
server_key_iv: "99ae639fce7b5a9901b52bbd8fef800d"
server_key_version: "encryptionkey@0"
I20260924 20:24:27.126848 8295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260924 20:24:27.129340 8310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:27.130761 8295 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:24:27.130935 8295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "fec25743e03f4c07badf221e662f6dbd"
format_stamp: "Formatted at 2026-09-24 20:24:27 on dist-test-slave-m1qq"
server_key: "46963523878af86bdce72cb6f79bc3bd"
server_key_iv: "99ae639fce7b5a9901b52bbd8fef800d"
server_key_version: "encryptionkey@0"
I20260924 20:24:27.131102 8295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:27.157317 8295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:27.160910 8295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:27.161156 8295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:27.168846 8295 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43681
I20260924 20:24:27.168872 8362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43681 every 8 connection(s)
I20260924 20:24:27.169870 8295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:27.171846 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8295
I20260924 20:24:27.171995 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:27.172356 19121 external_mini_cluster.cc:1428] Setting key 6cbc1f09ada0d241f6cd069cddb1e997
I20260924 20:24:27.173084 8363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:27.180191 8363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd: Bootstrap starting.
Sep 24 20:24:27 dist-test-slave-m1qq krb5kdc[8282](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:27.182658 8363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:27.183467 8363 log.cc:824] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:27.185865 8363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd: No bootstrap required, opened a new log
I20260924 20:24:27.188313 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:27.173735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58565 (local address 127.18.172.126:43681)
0924 20:24:27.174169 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:27.174188 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:27.174229 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:24:27.175122 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:27.175146 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:27.175588 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:27.175972 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:27.176664 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:27.178198 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:27.179127 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:27.179830 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:27.179992 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:27.182928 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:27.182933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:27.182946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:27.182973 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:27.185218 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:27.185826 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:27.185831 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:27.185836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:27.185945 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:27.186354 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:27.186357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:27.186358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:27.186779 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:27.186941 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:27.187315 (+ 374us) server_negotiation.cc:299] Negotiation successful
0924 20:24:27.187570 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260924 20:24:27.189880 8363 raft_consensus.cc:348] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } }
I20260924 20:24:27.190142 8363 raft_consensus.cc:374] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:27.190232 8363 raft_consensus.cc:729] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec25743e03f4c07badf221e662f6dbd, State: Initialized, Role: FOLLOWER
I20260924 20:24:27.190687 8363 consensus_queue.cc:260] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } }
I20260924 20:24:27.190846 8363 raft_consensus.cc:388] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:27.190943 8363 raft_consensus.cc:482] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:27.191083 8363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:27.191447 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:27.192327 8363 raft_consensus.cc:504] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } }
I20260924 20:24:27.192694 8363 leader_election.cc:302] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fec25743e03f4c07badf221e662f6dbd; no voters:
I20260924 20:24:27.193084 8363 leader_election.cc:288] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:27.193164 8372 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:27.193465 8372 raft_consensus.cc:686] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [term 1 LEADER]: Becoming Leader. State: Replica: fec25743e03f4c07badf221e662f6dbd, State: Running, Role: LEADER
I20260924 20:24:27.193830 8372 consensus_queue.cc:237] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } }
I20260924 20:24:27.194363 8363 sys_catalog.cc:564] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:27.194892 8372 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fec25743e03f4c07badf221e662f6dbd. Latest consensus state: current_term: 1 leader_uuid: "fec25743e03f4c07badf221e662f6dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } } }
I20260924 20:24:27.195114 8372 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:27.195364 8373 sys_catalog.cc:455] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fec25743e03f4c07badf221e662f6dbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fec25743e03f4c07badf221e662f6dbd" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43681 } } }
I20260924 20:24:27.195502 8373 sys_catalog.cc:458] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:27.195855 8382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:27.198817 8382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:27.204756 8382 catalog_manager.cc:1347] Generated new cluster ID: 189e09b397a741c6bf53c042818867d2
I20260924 20:24:27.204819 8382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:27.206465 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:27.193773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55745 (local address 127.18.172.126:43681)
0924 20:24:27.193884 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:27.193888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:27.197022 (+ 3134us) server_negotiation.cc:407] Connection header received
0924 20:24:27.197155 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:27.197158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:27.197206 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:27.197283 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:27.198714 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:27.199508 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:27.200658 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:27.200920 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:27.201003 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:27.202617 (+ 1614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:27.202620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:27.202623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:27.202643 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:27.204382 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:27.205033 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:27.205037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:27.205039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:27.205124 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:27.205503 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:27.205513 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:27.205515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:27.205770 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:27.205928 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:27.206190 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 20:24:27.206295 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":41}
I20260924 20:24:27.226452 8382 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:27.227654 8382 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:27.233174 8382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fec25743e03f4c07badf221e662f6dbd: Generated new TSK 0
I20260924 20:24:27.234010 8382 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:27.244547 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8295
W20260924 20:24:27.249600 8280 connection.cc:535] client connection to 127.18.172.126:43681 recv error: Network error: failed to read from TLS socket (remote: 127.18.172.126:43681): Connection reset by peer (error 104)
2026-09-24T20:24:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:27 dist-test-slave-m1qq krb5kdc[8394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:27 dist-test-slave-m1qq krb5kdc[8394](info): set up 2 sockets
Sep 24 20:24:27 dist-test-slave-m1qq krb5kdc[8394](info): commencing operation
krb5kdc: starting...
W20260924 20:24:28.296458 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:28 dist-test-slave-m1qq krb5kdc[8394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:28.422724 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:40911
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:42573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:40911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:28.533301 8407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:28.533623 8407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:28.533710 8407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:28.537763 8407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:28.537858 8407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:28.537897 8407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:28.537953 8407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:28.537994 8407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:28.543314 8407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:42573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:40911
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:40911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:28.544523 8407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:28.545635 8407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:28.554304 8415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:28.554320 8413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:28.554304 8412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:28.554625 8407 server_base.cc:1034] running on GCE node
I20260924 20:24:28.555409 8407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:28.556643 8407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:28.557865 8407 hybrid_clock.cc:648] HybridClock initialized: now 1790281468557840 us; error 36 us; skew 500 ppm
Sep 24 20:24:28 dist-test-slave-m1qq krb5kdc[8394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281468, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:28.560943 8407 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:28.561995 8407 webserver.cc:460] Webserver started at http://127.18.172.126:35653/ using document root <none> and password file <none>
I20260924 20:24:28.562594 8407 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:28.562702 8407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:28.562958 8407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:28.565205 8407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "cc021909b47a4d3aa17ac6dc19800685"
format_stamp: "Formatted at 2026-09-24 20:24:28 on dist-test-slave-m1qq"
server_key: "dbb87104ea3069b5cccea1e32c252534"
server_key_iv: "9f00ea76b674d239f842744314922e51"
server_key_version: "encryptionkey@0"
I20260924 20:24:28.565831 8407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "cc021909b47a4d3aa17ac6dc19800685"
format_stamp: "Formatted at 2026-09-24 20:24:28 on dist-test-slave-m1qq"
server_key: "dbb87104ea3069b5cccea1e32c252534"
server_key_iv: "9f00ea76b674d239f842744314922e51"
server_key_version: "encryptionkey@0"
I20260924 20:24:28.569918 8407 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260924 20:24:28.572517 8422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:28.573859 8407 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 20:24:28.574012 8407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "cc021909b47a4d3aa17ac6dc19800685"
format_stamp: "Formatted at 2026-09-24 20:24:28 on dist-test-slave-m1qq"
server_key: "dbb87104ea3069b5cccea1e32c252534"
server_key_iv: "9f00ea76b674d239f842744314922e51"
server_key_version: "encryptionkey@0"
I20260924 20:24:28.574134 8407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:28.613351 8407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:28.616648 8407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:28.616889 8407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:28.624903 8407 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:40911
I20260924 20:24:28.624936 8474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:40911 every 8 connection(s)
I20260924 20:24:28.625950 8407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:28.629133 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:28.629437 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8407
I20260924 20:24:28.629585 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:28.629885 19121 external_mini_cluster.cc:1428] Setting key f1925b2ec01a439fe6e48bc9060f0f1e
I20260924 20:24:28.637149 8475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685: Bootstrap starting.
I20260924 20:24:28.640142 8475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:28.640832 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:28.631373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56289 (local address 127.18.172.126:40911)
0924 20:24:28.631839 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:28.631860 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:28.631906 (+ 46us) server_negotiation.cc:407] Connection header received
0924 20:24:28.632797 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:28.632824 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:28.633291 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:28.633717 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:28.634146 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:28.635840 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:28.636759 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:28.637499 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:28.637679 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:28.638596 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:28.638604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:28.638630 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:28.639185 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:28.639257 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:28.639289 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:28.639436 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:28.639625 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 20:24:28.639768 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":125,"threads_started":1}
I20260924 20:24:28.641085 8475 log.cc:824] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:28.643317 8475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685: No bootstrap required, opened a new log
I20260924 20:24:28.643939 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:28.647909 8475 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } }
I20260924 20:24:28.648176 8475 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:28.648267 8475 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc021909b47a4d3aa17ac6dc19800685, State: Initialized, Role: FOLLOWER
I20260924 20:24:28.648784 8475 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } }
I20260924 20:24:28.648954 8475 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:28.649034 8475 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:28.649185 8475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:28.650404 8475 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } }
I20260924 20:24:28.650550 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:28.646066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56173 (local address 127.18.172.126:40911)
0924 20:24:28.646226 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:28.646233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:28.646302 (+ 69us) server_negotiation.cc:407] Connection header received
0924 20:24:28.646417 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:28.646420 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:28.646474 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:28.646559 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:28.646833 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:28.647698 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:28.648611 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:28.649019 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:28.649141 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:28.650045 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:28.650049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:28.650052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:28.650066 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:28.650095 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:28.650109 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:28.650220 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:28.650323 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 20:24:28.650364 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:24:28.650825 8475 leader_election.cc:302] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc021909b47a4d3aa17ac6dc19800685; no voters:
I20260924 20:24:28.651223 8475 leader_election.cc:288] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:28.651417 8485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:28.651643 8485 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [term 1 LEADER]: Becoming Leader. State: Replica: cc021909b47a4d3aa17ac6dc19800685, State: Running, Role: LEADER
I20260924 20:24:28.652021 8485 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } }
I20260924 20:24:28.652606 8475 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:28.654048 8486 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc021909b47a4d3aa17ac6dc19800685. Latest consensus state: current_term: 1 leader_uuid: "cc021909b47a4d3aa17ac6dc19800685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } } }
I20260924 20:24:28.654171 8486 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:28.654606 8494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:28.654505 8487 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc021909b47a4d3aa17ac6dc19800685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc021909b47a4d3aa17ac6dc19800685" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 40911 } } }
I20260924 20:24:28.655120 8487 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:28.657465 8494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:28.663297 8494 catalog_manager.cc:1347] Generated new cluster ID: c5df94b95b784751be5e3199df19c727
I20260924 20:24:28.663367 8494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:28.674974 8494 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:28.676272 8494 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:28.684470 8494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc021909b47a4d3aa17ac6dc19800685: Generated new TSK 0
I20260924 20:24:28.685339 8494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:28.696163 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8407
W20260924 20:24:28.701495 8392 connection.cc:535] client connection to 127.18.172.126:40911 recv error: Network error: failed to read from TLS socket (remote: 127.18.172.126:40911): Connection reset by peer (error 104)
2026-09-24T20:24:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1452 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:28 dist-test-slave-m1qq krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:28 dist-test-slave-m1qq krb5kdc[8506](info): set up 2 sockets
Sep 24 20:24:28 dist-test-slave-m1qq krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260924 20:24:29.748975 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:29 dist-test-slave-m1qq krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:29.880491 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44463
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:46383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:29.988870 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:29.989135 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:29.989177 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:29.993346 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:29.993422 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:29.993440 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:29.993458 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:29.993474 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:29.998823 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:46383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:29.999935 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:30.000960 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:24:30.009734 8519 server_base.cc:1034] running on GCE node
W20260924 20:24:30.009660 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:30.009701 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:30.009660 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:30.010527 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:30.011649 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:30.012858 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790281470012831 us; error 38 us; skew 500 ppm
Sep 24 20:24:30 dist-test-slave-m1qq krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281470, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:30.015544 8519 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:30.016395 8519 webserver.cc:460] Webserver started at http://127.18.172.126:46477/ using document root <none> and password file <none>
I20260924 20:24:30.016925 8519 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:30.016983 8519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:30.017146 8519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:30.019170 8519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "db3e6522160748a48643b7862aac54ca"
format_stamp: "Formatted at 2026-09-24 20:24:30 on dist-test-slave-m1qq"
server_key: "f710099c65b646c80455e1242eee5e25"
server_key_iv: "568d66f0e570f18bb099d5fa7e335e1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:30.019680 8519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "db3e6522160748a48643b7862aac54ca"
format_stamp: "Formatted at 2026-09-24 20:24:30 on dist-test-slave-m1qq"
server_key: "f710099c65b646c80455e1242eee5e25"
server_key_iv: "568d66f0e570f18bb099d5fa7e335e1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:30.023653 8519 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 20:24:30.026192 8534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:30.027686 8519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:30.027801 8519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "db3e6522160748a48643b7862aac54ca"
format_stamp: "Formatted at 2026-09-24 20:24:30 on dist-test-slave-m1qq"
server_key: "f710099c65b646c80455e1242eee5e25"
server_key_iv: "568d66f0e570f18bb099d5fa7e335e1c"
server_key_version: "encryptionkey@0"
I20260924 20:24:30.027895 8519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:30.032444 8519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:30.035418 8519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:30.035563 8519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:30.042719 8519 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44463
I20260924 20:24:30.042737 8586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44463 every 8 connection(s)
I20260924 20:24:30.043740 8519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:30.046502 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8519
I20260924 20:24:30.046664 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:30.046698 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:30.047046 19121 external_mini_cluster.cc:1428] Setting key dd3a23b64f9c6ce22e7fcb0e04c4740f
I20260924 20:24:30.053093 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:30.048455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39783 (local address 127.18.172.126:44463)
0924 20:24:30.048894 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:30.048911 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:30.048952 (+ 41us) server_negotiation.cc:407] Connection header received
0924 20:24:30.050076 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:30.050106 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:30.050486 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:30.050864 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:30.051701 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:30.051707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:30.051721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:30.051733 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:30.051794 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:30.051821 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:30.051962 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:30.052043 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 20:24:30.052205 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260924 20:24:30.053911 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca: Bootstrap starting.
I20260924 20:24:30.055896 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:30.056396 8587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:30.057139 8587 log.cc:824] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:30.059476 8590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:30.057920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33877 (local address 127.18.172.126:44463)
0924 20:24:30.058041 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:30.058045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:30.058228 (+ 183us) server_negotiation.cc:407] Connection header received
0924 20:24:30.058306 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:30.058309 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:30.058339 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:30.058417 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:30.059131 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:30.059134 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:30.059137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:30.059148 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:30.059181 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:30.059195 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:30.059287 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:30.059337 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 20:24:30.059371 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 20:24:30.059482 8587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca: No bootstrap required, opened a new log
I20260924 20:24:30.063130 8587 raft_consensus.cc:348] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } }
I20260924 20:24:30.063395 8587 raft_consensus.cc:374] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:30.063441 8587 raft_consensus.cc:729] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db3e6522160748a48643b7862aac54ca, State: Initialized, Role: FOLLOWER
I20260924 20:24:30.063796 8587 consensus_queue.cc:260] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } }
I20260924 20:24:30.063902 8587 raft_consensus.cc:388] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:30.063946 8587 raft_consensus.cc:482] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:30.064013 8587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:30.064832 8587 raft_consensus.cc:504] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } }
I20260924 20:24:30.065107 8587 leader_election.cc:302] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db3e6522160748a48643b7862aac54ca; no voters:
I20260924 20:24:30.065371 8587 leader_election.cc:288] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:30.065518 8597 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:30.065729 8597 raft_consensus.cc:686] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [term 1 LEADER]: Becoming Leader. State: Replica: db3e6522160748a48643b7862aac54ca, State: Running, Role: LEADER
I20260924 20:24:30.066063 8597 consensus_queue.cc:237] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } }
I20260924 20:24:30.066577 8587 sys_catalog.cc:564] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:30.067821 8599 sys_catalog.cc:455] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db3e6522160748a48643b7862aac54ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } } }
I20260924 20:24:30.067814 8598 sys_catalog.cc:455] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader db3e6522160748a48643b7862aac54ca. Latest consensus state: current_term: 1 leader_uuid: "db3e6522160748a48643b7862aac54ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db3e6522160748a48643b7862aac54ca" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44463 } } }
I20260924 20:24:30.067965 8598 sys_catalog.cc:458] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:30.067965 8599 sys_catalog.cc:458] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:30.068427 8606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:30.071233 8606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:30.076825 8606 catalog_manager.cc:1347] Generated new cluster ID: 5f0bb0bd64014b80879efc00c005b8f8
I20260924 20:24:30.076915 8606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:30.090579 8606 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:30.090994 8606 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:30.102097 8606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db3e6522160748a48643b7862aac54ca: Generated new TSK 0
I20260924 20:24:30.102808 8606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:30.110760 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8519
2026-09-24T20:24:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1413 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:30 dist-test-slave-m1qq krb5kdc[8618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:30 dist-test-slave-m1qq krb5kdc[8618](info): set up 2 sockets
Sep 24 20:24:30 dist-test-slave-m1qq krb5kdc[8618](info): commencing operation
krb5kdc: starting...
W20260924 20:24:31.180513 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:31.310333 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44537
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:34091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:44537
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:31.423075 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:31.423415 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:31.423521 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:31.427687 8631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:31.427795 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:31.427836 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:31.427897 8631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:31.427937 8631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:31.433344 8631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:34091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:44537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:44537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:31.434516 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:31.435617 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:31.444321 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:31.444401 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:31.444482 8631 server_base.cc:1034] running on GCE node
W20260924 20:24:31.444331 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:31.445152 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:31.446357 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:31.447541 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790281471447529 us; error 25 us; skew 500 ppm
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281471, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:31.450423 8631 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:31.451377 8631 webserver.cc:460] Webserver started at http://127.18.172.126:46675/ using document root <none> and password file <none>
I20260924 20:24:31.451952 8631 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:31.452044 8631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:31.452282 8631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:31.454337 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "d23cf425017b40c19a1bc6ed4c693533"
format_stamp: "Formatted at 2026-09-24 20:24:31 on dist-test-slave-m1qq"
server_key: "58884d038c2808504038b34b0380f168"
server_key_iv: "852517ce0b19a1a797b4d70c2a993b47"
server_key_version: "encryptionkey@0"
I20260924 20:24:31.454918 8631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "d23cf425017b40c19a1bc6ed4c693533"
format_stamp: "Formatted at 2026-09-24 20:24:31 on dist-test-slave-m1qq"
server_key: "58884d038c2808504038b34b0380f168"
server_key_iv: "852517ce0b19a1a797b4d70c2a993b47"
server_key_version: "encryptionkey@0"
I20260924 20:24:31.458705 8631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:31.461210 8646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:31.462505 8631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260924 20:24:31.462752 8631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "d23cf425017b40c19a1bc6ed4c693533"
format_stamp: "Formatted at 2026-09-24 20:24:31 on dist-test-slave-m1qq"
server_key: "58884d038c2808504038b34b0380f168"
server_key_iv: "852517ce0b19a1a797b4d70c2a993b47"
server_key_version: "encryptionkey@0"
I20260924 20:24:31.462873 8631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:31.487181 8631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:31.490869 8631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:31.491161 8631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:31.498904 8631 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:44537
I20260924 20:24:31.498930 8698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:44537 every 8 connection(s)
I20260924 20:24:31.499910 8631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:31.503124 8699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:31.507197 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8631
I20260924 20:24:31.507354 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:31.507616 19121 external_mini_cluster.cc:1428] Setting key 72a26729a602227a6a12996129aadb42
I20260924 20:24:31.509930 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533: Bootstrap starting.
I20260924 20:24:31.512495 8699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:31.513478 8699 log.cc:824] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:31.515971 8699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533: No bootstrap required, opened a new log
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281471, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:31.519774 8699 raft_consensus.cc:348] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } }
I20260924 20:24:31.519989 8699 raft_consensus.cc:374] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:31.520051 8699 raft_consensus.cc:729] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d23cf425017b40c19a1bc6ed4c693533, State: Initialized, Role: FOLLOWER
I20260924 20:24:31.520529 8699 consensus_queue.cc:260] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } }
I20260924 20:24:31.520651 8699 raft_consensus.cc:388] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:31.520741 8699 raft_consensus.cc:482] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:31.520861 8699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:31.521755 8699 raft_consensus.cc:504] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } }
I20260924 20:24:31.522106 8699 leader_election.cc:302] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d23cf425017b40c19a1bc6ed4c693533; no voters:
I20260924 20:24:31.522432 8699 leader_election.cc:288] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:31.522536 8704 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:31.522773 8704 raft_consensus.cc:686] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [term 1 LEADER]: Becoming Leader. State: Replica: d23cf425017b40c19a1bc6ed4c693533, State: Running, Role: LEADER
I20260924 20:24:31.522836 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:31.509077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48861 (local address 127.18.172.126:44537)
0924 20:24:31.509481 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:31.509496 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:31.509533 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:24:31.510272 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:31.510299 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:31.510662 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:31.510953 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:31.511904 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:31.513047 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:31.514166 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:31.514710 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:31.514841 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:31.517769 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:31.517774 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:31.517791 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:31.517821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:31.519614 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:31.520120 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:31.520128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:31.520136 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:31.520258 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:31.520557 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:31.520560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:31.520562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:31.521087 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:31.521313 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:31.521552 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 20:24:31.521867 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":113,"threads_started":1}
I20260924 20:24:31.523195 8704 consensus_queue.cc:237] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } }
I20260924 20:24:31.523838 8699 sys_catalog.cc:564] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:31.524379 8704 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d23cf425017b40c19a1bc6ed4c693533. Latest consensus state: current_term: 1 leader_uuid: "d23cf425017b40c19a1bc6ed4c693533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } } }
I20260924 20:24:31.525106 8704 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:31.524999 8705 sys_catalog.cc:455] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d23cf425017b40c19a1bc6ed4c693533" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d23cf425017b40c19a1bc6ed4c693533" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 44537 } } }
I20260924 20:24:31.525410 8705 sys_catalog.cc:458] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:31.525808 8713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:31.526911 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:31.529225 8713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:31.536571 8713 catalog_manager.cc:1347] Generated new cluster ID: 3ae70392c0fa4b9594b2a6347f14d170
I20260924 20:24:31.536674 8713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:31.538125 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:31.529179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43987 (local address 127.18.172.126:44537)
0924 20:24:31.529311 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:31.529315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:31.529538 (+ 223us) server_negotiation.cc:407] Connection header received
0924 20:24:31.529696 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:31.529702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:31.529749 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:31.529844 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:31.530582 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:31.531272 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:31.532486 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:31.532811 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:31.532922 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:31.534806 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:31.534811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:31.534814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:31.534841 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:31.536401 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:31.536908 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:31.536913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:31.536915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:31.537003 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:31.537277 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:31.537281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:31.537283 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:31.537470 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:31.537603 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:31.537837 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 20:24:31.537950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 20:24:31.557175 8713 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:31.558490 8713 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:31.566039 8713 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d23cf425017b40c19a1bc6ed4c693533: Generated new TSK 0
I20260924 20:24:31.566915 8713 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:31.576486 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8631
2026-09-24T20:24:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1466 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8730](info): set up 2 sockets
Sep 24 20:24:31 dist-test-slave-m1qq krb5kdc[8730](info): commencing operation
krb5kdc: starting...
W20260924 20:24:32.697273 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:32 dist-test-slave-m1qq krb5kdc[8730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:32.826538 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:33171
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:33171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:32.942135 8743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:32.942468 8743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:32.942549 8743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:32.946645 8743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:32.946763 8743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:32.946801 8743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:32.946837 8743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:32.946885 8743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:32.952380 8743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:33171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:33171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:32.953533 8743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:32.954613 8743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 20:24:32.963554 8743 server_base.cc:1034] running on GCE node
W20260924 20:24:32.963490 8748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:32.963523 8751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:32.963502 8749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:32.964316 8743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:32.965480 8743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:32.966668 8743 hybrid_clock.cc:648] HybridClock initialized: now 1790281472966658 us; error 27 us; skew 500 ppm
Sep 24 20:24:32 dist-test-slave-m1qq krb5kdc[8730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281472, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:32.973956 8743 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:32.974951 8743 webserver.cc:460] Webserver started at http://127.18.172.126:41749/ using document root <none> and password file <none>
I20260924 20:24:32.975572 8743 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:32.975668 8743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:32.975908 8743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:32.978019 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "a29193ec946243a39f7ebd98d419b3ce"
format_stamp: "Formatted at 2026-09-24 20:24:32 on dist-test-slave-m1qq"
server_key: "77841078fa196f78ebcf1d42d3067c9a"
server_key_iv: "29bc50831dbb1d662c26a5273092839b"
server_key_version: "encryptionkey@0"
I20260924 20:24:32.978616 8743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "a29193ec946243a39f7ebd98d419b3ce"
format_stamp: "Formatted at 2026-09-24 20:24:32 on dist-test-slave-m1qq"
server_key: "77841078fa196f78ebcf1d42d3067c9a"
server_key_iv: "29bc50831dbb1d662c26a5273092839b"
server_key_version: "encryptionkey@0"
I20260924 20:24:32.982466 8743 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:24:32.985136 8758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:32.986824 8743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 20:24:32.986974 8743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "a29193ec946243a39f7ebd98d419b3ce"
format_stamp: "Formatted at 2026-09-24 20:24:32 on dist-test-slave-m1qq"
server_key: "77841078fa196f78ebcf1d42d3067c9a"
server_key_iv: "29bc50831dbb1d662c26a5273092839b"
server_key_version: "encryptionkey@0"
I20260924 20:24:32.987174 8743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:33.001579 8743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:33.004999 8743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:33.005210 8743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:33.012723 8743 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:33171
I20260924 20:24:33.012748 8810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:33171 every 8 connection(s)
I20260924 20:24:33.013703 8743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:33.016629 8811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:33.022212 8811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce: Bootstrap starting.
I20260924 20:24:33.023307 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8743
I20260924 20:24:33.023450 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:33.023767 19121 external_mini_cluster.cc:1428] Setting key 5dae3a52d0334552c1e53768f92c56b0
I20260924 20:24:33.025218 8811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:33.025954 8811 log.cc:824] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:33.028141 8811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce: No bootstrap required, opened a new log
I20260924 20:24:33.032106 8811 raft_consensus.cc:348] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } }
I20260924 20:24:33.032433 8811 raft_consensus.cc:374] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:33.032531 8811 raft_consensus.cc:729] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a29193ec946243a39f7ebd98d419b3ce, State: Initialized, Role: FOLLOWER
I20260924 20:24:33.032979 8811 consensus_queue.cc:260] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } }
I20260924 20:24:33.033104 8811 raft_consensus.cc:388] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:33.033195 8811 raft_consensus.cc:482] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:33.033336 8811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:33.033488 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:33.024960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36129 (local address 127.18.172.126:33171)
0924 20:24:33.025456 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:33.025472 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:33.025507 (+ 35us) server_negotiation.cc:407] Connection header received
0924 20:24:33.026212 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:33.026231 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:33.026691 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:33.027102 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:33.027509 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:33.028545 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:33.029552 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:33.030043 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:33.030210 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:33.031273 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:33.031279 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:33.031298 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:33.031842 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:33.031898 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:33.031924 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:33.032106 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:33.032252 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:24:33.032464 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":135,"threads_started":1}
I20260924 20:24:33.034314 8811 raft_consensus.cc:504] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } }
I20260924 20:24:33.034688 8811 leader_election.cc:302] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a29193ec946243a39f7ebd98d419b3ce; no voters:
I20260924 20:24:33.035039 8811 leader_election.cc:288] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:33.035218 8816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:33.035511 8816 raft_consensus.cc:686] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [term 1 LEADER]: Becoming Leader. State: Replica: a29193ec946243a39f7ebd98d419b3ce, State: Running, Role: LEADER
I20260924 20:24:33.035823 8816 consensus_queue.cc:237] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } }
I20260924 20:24:33.036399 8811 sys_catalog.cc:564] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:33.036679 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:33.037530 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a29193ec946243a39f7ebd98d419b3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } } }
I20260924 20:24:33.037665 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:33.037911 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader a29193ec946243a39f7ebd98d419b3ce. Latest consensus state: current_term: 1 leader_uuid: "a29193ec946243a39f7ebd98d419b3ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a29193ec946243a39f7ebd98d419b3ce" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 33171 } } }
I20260924 20:24:33.037986 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:33.038265 8825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:33.042132 8825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:33.043967 8814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:33.039085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49943 (local address 127.18.172.126:33171)
0924 20:24:33.039173 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:33.039177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:33.039416 (+ 239us) server_negotiation.cc:407] Connection header received
0924 20:24:33.039556 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:33.039559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:33.039606 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:33.039691 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:33.039959 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:33.040669 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:33.041794 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:33.042126 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:33.042234 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:33.043429 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:33.043435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:33.043481 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:33.043494 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:33.043526 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:33.043541 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:33.043621 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:33.043767 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:24:33.043833 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 20:24:33.048414 8825 catalog_manager.cc:1347] Generated new cluster ID: ec27be78808a41128cb4545e44cbfb20
I20260924 20:24:33.048483 8825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:33.078068 8825 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:33.079380 8825 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:33.088048 8825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a29193ec946243a39f7ebd98d419b3ce: Generated new TSK 0
I20260924 20:24:33.088923 8825 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:33.100517 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8743
2026-09-24T20:24:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1524 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:33 dist-test-slave-m1qq krb5kdc[8842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:33 dist-test-slave-m1qq krb5kdc[8842](info): set up 2 sockets
Sep 24 20:24:33 dist-test-slave-m1qq krb5kdc[8842](info): commencing operation
krb5kdc: starting...
W20260924 20:24:34.188570 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:34 dist-test-slave-m1qq krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:34.318747 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38479
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:35567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:38479
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:34.439634 8855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:34.439937 8855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:34.440019 8855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:34.444439 8855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:34.444550 8855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:34.444593 8855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:34.444655 8855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:34.444695 8855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:34.450112 8855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:35567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:38479
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:38479
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:34.451378 8855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:34.452706 8855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:34.462386 8861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:34.462585 8855 server_base.cc:1034] running on GCE node
W20260924 20:24:34.462384 8860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:34.462344 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:34.463379 8855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:34.464759 8855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:34.465960 8855 hybrid_clock.cc:648] HybridClock initialized: now 1790281474465948 us; error 30 us; skew 500 ppm
Sep 24 20:24:34 dist-test-slave-m1qq krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281474, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:34.468782 8855 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:34.469637 8855 webserver.cc:460] Webserver started at http://127.18.172.126:32939/ using document root <none> and password file <none>
I20260924 20:24:34.470198 8855 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:34.470289 8855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:34.470516 8855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:34.472643 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "26509a267054443db2c90b3b155a761c"
format_stamp: "Formatted at 2026-09-24 20:24:34 on dist-test-slave-m1qq"
server_key: "1769a63b573aec3f97e2f7b22baa291c"
server_key_iv: "fd0b8a9f2eaa84333c63e4678cfd31a5"
server_key_version: "encryptionkey@0"
I20260924 20:24:34.473197 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "26509a267054443db2c90b3b155a761c"
format_stamp: "Formatted at 2026-09-24 20:24:34 on dist-test-slave-m1qq"
server_key: "1769a63b573aec3f97e2f7b22baa291c"
server_key_iv: "fd0b8a9f2eaa84333c63e4678cfd31a5"
server_key_version: "encryptionkey@0"
I20260924 20:24:34.477401 8855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 20:24:34.480417 8870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:34.481735 8855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 20:24:34.481984 8855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "26509a267054443db2c90b3b155a761c"
format_stamp: "Formatted at 2026-09-24 20:24:34 on dist-test-slave-m1qq"
server_key: "1769a63b573aec3f97e2f7b22baa291c"
server_key_iv: "fd0b8a9f2eaa84333c63e4678cfd31a5"
server_key_version: "encryptionkey@0"
I20260924 20:24:34.482133 8855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:34.486969 8855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:34.490224 8855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:34.490432 8855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:34.498045 8855 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:38479
I20260924 20:24:34.498126 8922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:38479 every 8 connection(s)
I20260924 20:24:34.499178 8855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:34.502048 8923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:34.505607 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8855
I20260924 20:24:34.505755 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:34.506052 19121 external_mini_cluster.cc:1428] Setting key 3d438c117d10c615bdc8dd9801800336
I20260924 20:24:34.508157 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c: Bootstrap starting.
I20260924 20:24:34.510742 8923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:34.511425 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:34.507485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49671 (local address 127.18.172.126:38479)
0924 20:24:34.507916 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:34.507932 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:34.507968 (+ 36us) server_negotiation.cc:407] Connection header received
0924 20:24:34.508656 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:34.508674 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:34.508971 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:34.509244 (+ 273us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:34.510093 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:34.510097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:34.510108 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:34.510118 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:34.510171 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:34.510190 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:34.510276 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:34.510422 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 20:24:34.510606 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":125,"threads_started":1}
I20260924 20:24:34.511538 8923 log.cc:824] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:34.513777 8923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c: No bootstrap required, opened a new log
I20260924 20:24:34.514396 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:34.518765 8926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:34.516735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50967 (local address 127.18.172.126:38479)
0924 20:24:34.516864 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:34.516869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:34.517142 (+ 273us) server_negotiation.cc:407] Connection header received
0924 20:24:34.517313 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:34.517315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:34.517348 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:34.517463 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:34.518291 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:34.518294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:34.518296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 20:24:34.518305 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:34.518328 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 20:24:34.518338 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:34.518445 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:34.518560 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 20:24:34.518592 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 20:24:34.519368 8923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } }
I20260924 20:24:34.519670 8923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 20:24:34.519745 8923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26509a267054443db2c90b3b155a761c, State: Initialized, Role: FOLLOWER
I20260924 20:24:34.520210 8923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } }
I20260924 20:24:34.520349 8923 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 20:24:34.520439 8923 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 20:24:34.520550 8923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:34.521562 8923 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } }
I20260924 20:24:34.521917 8923 leader_election.cc:302] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26509a267054443db2c90b3b155a761c; no voters:
I20260924 20:24:34.522226 8923 leader_election.cc:288] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 20:24:34.522342 8933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:34.522576 8933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [term 1 LEADER]: Becoming Leader. State: Replica: 26509a267054443db2c90b3b155a761c, State: Running, Role: LEADER
I20260924 20:24:34.522921 8933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } }
I20260924 20:24:34.523677 8923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:34.524832 8934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26509a267054443db2c90b3b155a761c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } } }
I20260924 20:24:34.524969 8934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:34.525250 8935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26509a267054443db2c90b3b155a761c. Latest consensus state: current_term: 1 leader_uuid: "26509a267054443db2c90b3b155a761c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26509a267054443db2c90b3b155a761c" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 38479 } } }
I20260924 20:24:34.525424 8935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:34.525362 8942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:34.528554 8942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:34.534126 8942 catalog_manager.cc:1347] Generated new cluster ID: 4cebaccb72154603a9bec426b34a4e41
I20260924 20:24:34.534211 8942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:34.553741 8942 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:34.554164 8942 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:34.561728 8942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26509a267054443db2c90b3b155a761c: Generated new TSK 0
I20260924 20:24:34.562631 8942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 20:24:34.576774 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8855
2026-09-24T20:24:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1475 ms)
[----------] 18 tests from AuthTokenIssuingTest (26190 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:34 dist-test-slave-m1qq krb5kdc[8954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:34 dist-test-slave-m1qq krb5kdc[8954](info): set up 2 sockets
Sep 24 20:24:34 dist-test-slave-m1qq krb5kdc[8954](info): commencing operation
krb5kdc: starting...
W20260924 20:24:35.704944 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:35 dist-test-slave-m1qq krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:35.836256 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39299
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:35.948772 8967 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:35.949069 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:35.949092 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:35.949182 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:35.953408 8967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:35.953472 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:35.953491 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:35.953508 8967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:35.953523 8967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:35.959133 8967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:39299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:35.960274 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:35.961309 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:35.969930 8972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:35.970084 8967 server_base.cc:1034] running on GCE node
W20260924 20:24:35.970016 8973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:35.970007 8975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:35.970780 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:35.971906 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:35.973097 8967 hybrid_clock.cc:648] HybridClock initialized: now 1790281475973059 us; error 54 us; skew 500 ppm
Sep 24 20:24:35 dist-test-slave-m1qq krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:35.975934 8967 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:35.976800 8967 webserver.cc:460] Webserver started at http://127.18.172.126:33045/ using document root <none> and password file <none>
I20260924 20:24:35.977310 8967 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:35.977370 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:35.977530 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:35.979597 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "ff1a2385169e4aba90c7d5883e2cdf41"
format_stamp: "Formatted at 2026-09-24 20:24:35 on dist-test-slave-m1qq"
server_key: "80ef432a9980b35fa2b63268b2babc76"
server_key_iv: "0a61543425506e25977c38f9d7d1a0fb"
server_key_version: "encryptionkey@0"
I20260924 20:24:35.980110 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "ff1a2385169e4aba90c7d5883e2cdf41"
format_stamp: "Formatted at 2026-09-24 20:24:35 on dist-test-slave-m1qq"
server_key: "80ef432a9980b35fa2b63268b2babc76"
server_key_iv: "0a61543425506e25977c38f9d7d1a0fb"
server_key_version: "encryptionkey@0"
I20260924 20:24:35.983884 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 20:24:35.986332 8982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:35.987637 8967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260924 20:24:35.987769 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "ff1a2385169e4aba90c7d5883e2cdf41"
format_stamp: "Formatted at 2026-09-24 20:24:35 on dist-test-slave-m1qq"
server_key: "80ef432a9980b35fa2b63268b2babc76"
server_key_iv: "0a61543425506e25977c38f9d7d1a0fb"
server_key_version: "encryptionkey@0"
I20260924 20:24:35.987885 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:36.012213 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:36.015664 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:36.015919 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:36.023236 8967 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:39299
I20260924 20:24:36.023689 9034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:39299 every 8 connection(s)
I20260924 20:24:36.024260 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:36.027163 9035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:36.030941 9035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.033209 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 8967
I20260924 20:24:36.033354 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:36.033911 19121 external_mini_cluster.cc:1428] Setting key aac56900b3aa9975889c18429890965c
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.125@KRBTEST.COM" created.
I20260924 20:24:36.046420 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.033276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39299 (local address 127.0.0.1:42032)
0924 20:24:36.033901 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.033953 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.034792 (+ 839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.035519 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.035525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.035979 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.036304 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.036329 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.037498 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.037503 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.038250 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.038259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.038928 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.038932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.039057 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.039702 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.039719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.041176 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.043235 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.043244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.043255 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.043491 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.043830 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.043833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.043835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.043953 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.044620 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.044627 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.044773 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.044946 (+ 173us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.045351 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":114,"thread_start_us":341,"threads_started":1}
I20260924 20:24:36.046420 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.033276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.18.172.126:39299)
0924 20:24:36.033927 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.033946 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.034075 (+ 129us) server_negotiation.cc:407] Connection header received
0924 20:24:36.035127 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.035132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.035325 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.035523 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.036492 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.037370 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.038382 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.038765 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.038942 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.041336 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.041343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.041356 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.041377 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.043111 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.043608 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.043613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.043615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.043727 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.044056 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.044059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.044061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.044499 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.044642 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.045017 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.045291 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":536,"spinlock_wait_cycles":30080,"thread_start_us":219,"threads_started":1}
I20260924 20:24:36.048905 9035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.049701 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.049437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58340)
0924 20:24:36.049557 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.049637 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:24:36.050011 8985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
I20260924 20:24:36.050904 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.050692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58344)
0924 20:24:36.050800 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.050845 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:24:36.051169 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:46231: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
I20260924 20:24:36.063602 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:46231
--webserver_interface=127.18.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:24:36.100437 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } attempt: 1
I20260924 20:24:36.101356 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.101032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58358)
0924 20:24:36.101196 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.101263 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 20:24:36.102401 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.102024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58372)
0924 20:24:36.102199 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.102339 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260924 20:24:36.102654 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:46231: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
I20260924 20:24:36.170886 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } attempt: 2
I20260924 20:24:36.171744 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.171447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58388)
0924 20:24:36.171597 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.171666 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:24:36.172605 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.172319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:58390)
0924 20:24:36.172469 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.172526 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 20:24:36.172796 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:46231: Network error: Client connection negotiation failed: client connection to 127.18.172.125:46231: connect: Connection refused (error 111)
W20260924 20:24:36.179371 9042 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:36.179656 9042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:36.179706 9042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:36.179812 9042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:36.183939 9042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:36.184001 9042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:36.184021 9042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:36.184075 9042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:36.184101 9042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:36.189591 9042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:36.190948 9042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:36.192126 9042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:36.200881 9047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:36.200977 9050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:36.200892 9048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:36.200948 9042 server_base.cc:1034] running on GCE node
I20260924 20:24:36.201687 9042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:36.202857 9042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:36.204046 9042 hybrid_clock.cc:648] HybridClock initialized: now 1790281476204031 us; error 32 us; skew 500 ppm
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:36.207093 9042 init.cc:377] Logged in from keytab as kudu/127.18.172.125@KRBTEST.COM (short username kudu)
I20260924 20:24:36.208062 9042 webserver.cc:460] Webserver started at http://127.18.172.125:37933/ using document root <none> and password file <none>
I20260924 20:24:36.208633 9042 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:36.208716 9042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:36.208949 9042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:36.211102 9042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data/instance:
uuid: "28c4e380c30b47ef96d1fc6a38f06f20"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "6af4161b759055cea9e72a621448f42a"
server_key_iv: "40ed45b81c71cd667e77c6f6402d10d6"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.211673 9042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal/instance:
uuid: "28c4e380c30b47ef96d1fc6a38f06f20"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "6af4161b759055cea9e72a621448f42a"
server_key_iv: "40ed45b81c71cd667e77c6f6402d10d6"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.215466 9042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 20:24:36.218164 9057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:36.219515 9042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 20:24:36.219672 9042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "28c4e380c30b47ef96d1fc6a38f06f20"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "6af4161b759055cea9e72a621448f42a"
server_key_iv: "40ed45b81c71cd667e77c6f6402d10d6"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.219811 9042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:36.250625 9042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:36.254072 9042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:36.254392 9042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:36.261619 9042 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.125:46231
I20260924 20:24:36.261920 9109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.125:46231 every 8 connection(s)
I20260924 20:24:36.262638 9042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
I20260924 20:24:36.263962 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } attempt: 3
I20260924 20:24:36.265887 9110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:36.270093 9110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.270754 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 9042
I20260924 20:24:36.270890 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-1/wal/instance
I20260924 20:24:36.271203 19121 external_mini_cluster.cc:1428] Setting key 40de3c315fba7fe483cd00483e62de00
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:36.277598 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.264519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:36184)
0924 20:24:36.264695 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.264714 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.264818 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.267063 (+ 2245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.267066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.267281 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.267423 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.267432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.268778 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.268781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.269650 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.269665 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.270383 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.270387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.270497 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.271105 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.271120 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.272817 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.275402 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.275408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.275414 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.275695 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.276159 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.276165 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.276169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.276283 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.277053 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.277057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.277186 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.277306 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.277466 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260924 20:24:36.279016 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.265154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.18.172.125:46231)
0924 20:24:36.265556 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.265573 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.265622 (+ 49us) server_negotiation.cc:407] Connection header received
0924 20:24:36.266407 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.266427 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.266746 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.267092 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.267612 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.268634 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.269784 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.270249 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.270367 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.273060 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.273066 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.273081 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.273111 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.275280 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.275809 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.275814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.275820 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.275988 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.276410 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.276413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.276415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.276942 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.277085 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.277521 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.277858 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":131,"threads_started":1}
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.18.172.124@KRBTEST.COM; defaulting to no policy
I20260924 20:24:36.281262 9035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.282081 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.281829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60290)
0924 20:24:36.281955 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.282012 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 20:24:36.283138 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.282804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60294)
0924 20:24:36.282989 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.283067 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 20:24:36.283289 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.124@KRBTEST.COM" created.
I20260924 20:24:36.284562 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.272125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.18.172.126:39299)
0924 20:24:36.272256 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.272260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.272608 (+ 348us) server_negotiation.cc:407] Connection header received
0924 20:24:36.272819 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.272825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.272901 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.272988 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.273848 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.274501 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.275680 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.276598 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.276708 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.280970 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.280973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.280975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.280997 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.282612 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.283219 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.283224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.283226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.283319 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.283682 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.283685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.283686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.283868 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.283948 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.284300 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.284418 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 20:24:36.284605 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.272125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39299 (local address 127.0.0.1:43412)
0924 20:24:36.272489 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.272536 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.272698 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.273046 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.273051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.273514 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.273697 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.273704 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.274624 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.274629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.275570 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.275577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.276735 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.276742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.276891 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.279185 (+ 2294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.279206 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.280738 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.282739 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.282747 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.282751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.283086 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.283462 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.283465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.283470 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.283570 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.283962 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.283967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.284070 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.284216 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.284418 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260924 20:24:36.285624 9110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:36.294848 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.286130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:36194)
0924 20:24:36.286216 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.286230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.286301 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.286557 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.286559 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.286688 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.286778 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.286783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.287841 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.287843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.288358 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.288365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.288824 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.288827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.288957 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.289585 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.289597 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.290856 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.293346 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.293349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.293350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.293547 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.293842 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.293844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.293845 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.293922 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.294400 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.294406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.294515 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.294608 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.294720 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":14}
I20260924 20:24:36.294996 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.286200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36194 (local address 127.18.172.125:46231)
0924 20:24:36.286326 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.286332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.286347 (+ 15us) server_negotiation.cc:407] Connection header received
0924 20:24:36.286395 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.286399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.286447 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.286535 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.286892 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.287736 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.288489 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.288720 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.288826 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.291056 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.291060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.291063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.291092 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.293158 (+ 2066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.293664 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.293667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.293668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.293745 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.294013 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.294015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.294016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.294197 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.294281 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.294692 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.294797 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":48}
I20260924 20:24:36.295811 9110 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.296895 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.296451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60298)
0924 20:24:36.296760 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.296839 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224}
W20260924 20:24:36.297219 9060 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
I20260924 20:24:36.298151 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.297959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60300)
0924 20:24:36.298062 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.298106 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:24:36.298316 9110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
I20260924 20:24:36.302838 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:44983
--webserver_interface=127.18.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:33167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:24:36.314528 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 1
I20260924 20:24:36.315408 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.315128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60308)
0924 20:24:36.315270 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.315333 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 20:24:36.316136 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.315963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60318)
0924 20:24:36.316051 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.316092 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 20:24:36.316320 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
I20260924 20:24:36.347625 9110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 1
I20260924 20:24:36.348529 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.348221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60334)
0924 20:24:36.348386 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.348452 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 20:24:36.349273 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.349059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60338)
0924 20:24:36.349171 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.349221 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260924 20:24:36.349393 9110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
I20260924 20:24:36.391500 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 2
I20260924 20:24:36.392514 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.392124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60342)
0924 20:24:36.392310 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.392434 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 20:24:36.393283 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.393068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60348)
0924 20:24:36.393177 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.393229 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:24:36.393425 9035 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
I20260924 20:24:36.417564 9110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 2
I20260924 20:24:36.418476 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.418161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60358)
0924 20:24:36.418324 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.418396 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:24:36.418496 9117 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:36.418865 9117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:36.418934 9117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:36.419047 9117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 20:24:36.419238 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.418992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60370)
0924 20:24:36.419119 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.419174 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:24:36.419422 9110 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:44983: Network error: Client connection negotiation failed: client connection to 127.18.172.124:44983: connect: Connection refused (error 111)
W20260924 20:24:36.423259 9117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:36.423339 9117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:36.423380 9117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:36.423413 9117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:36.423448 9117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:36.429013 9117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:33167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:39299,127.18.172.125:46231,127.18.172.124:44983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:44983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:36.430114 9117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:36.431336 9117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:36.440253 9125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:36.440259 9122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:36.440296 9123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:36.440419 9117 server_base.cc:1034] running on GCE node
I20260924 20:24:36.441162 9117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:36.442343 9117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:36.443542 9117 hybrid_clock.cc:648] HybridClock initialized: now 1790281476443521 us; error 33 us; skew 500 ppm
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:36.446422 9117 init.cc:377] Logged in from keytab as kudu/127.18.172.124@KRBTEST.COM (short username kudu)
I20260924 20:24:36.447343 9117 webserver.cc:460] Webserver started at http://127.18.172.124:38303/ using document root <none> and password file <none>
I20260924 20:24:36.447875 9117 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:36.447932 9117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:36.448096 9117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:36.450094 9117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data/instance:
uuid: "331d5369042d45c59648c96de2bb69aa"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "347a5cd682be108666f7e6c80bcf5584"
server_key_iv: "0d39ec9d27fdf60e1bfc00b78efabc92"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.450631 9117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal/instance:
uuid: "331d5369042d45c59648c96de2bb69aa"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "347a5cd682be108666f7e6c80bcf5584"
server_key_iv: "0d39ec9d27fdf60e1bfc00b78efabc92"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.454454 9117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 20:24:36.456972 9132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:36.458411 9117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 20:24:36.458556 9117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "331d5369042d45c59648c96de2bb69aa"
format_stamp: "Formatted at 2026-09-24 20:24:36 on dist-test-slave-m1qq"
server_key: "347a5cd682be108666f7e6c80bcf5584"
server_key_iv: "0d39ec9d27fdf60e1bfc00b78efabc92"
server_key_version: "encryptionkey@0"
I20260924 20:24:36.458669 9117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:36.471709 9117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:36.474977 9117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:36.475236 9117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:36.482803 9117 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.124:44983
I20260924 20:24:36.483235 9184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.124:44983 every 8 connection(s)
I20260924 20:24:36.483860 9117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
I20260924 20:24:36.486670 9185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:36.489295 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 9117
I20260924 20:24:36.489440 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790281188136637-19121-0/minicluster-data/master-2/wal/instance
I20260924 20:24:36.489729 19121 external_mini_cluster.cc:1428] Setting key 1e5076fca8943aac4cddcce221e57fae
I20260924 20:24:36.491201 9185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.492583 9035 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 3
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
I20260924 20:24:36.504634 9037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.493013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60384)
0924 20:24:36.493282 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.493296 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.493394 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.494305 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.494308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.494517 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.494630 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.494638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.496023 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.496027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.496833 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.496844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.497551 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.497631 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.498275 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.498291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.499917 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.502593 (+ 2676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.502600 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.502601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.502839 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.503272 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.503278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.503280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.503371 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.504127 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.504130 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.504214 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.504314 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.504458 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260924 20:24:36.504766 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.493174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43424 (local address 127.18.172.126:39299)
0924 20:24:36.493521 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.493525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.493536 (+ 11us) server_negotiation.cc:407] Connection header received
0924 20:24:36.493702 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.493705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.493749 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.493797 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.494936 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.495707 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497245 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.497509 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497604 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.500786 (+ 3182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.500789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.500792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.500822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.502394 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.503091 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.503097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.503102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.503191 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.503596 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.503599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.503601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.503822 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.503943 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.504411 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.504562 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288}
I20260924 20:24:36.505725 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.493041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:39299 (local address 127.0.0.1:43424)
0924 20:24:36.493436 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.493484 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.493579 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.494022 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.494028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.494483 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.494793 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.494806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.495859 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.495865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.497126 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.497135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.497777 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.497908 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.498564 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.498590 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.500615 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.502512 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.502524 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.502536 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.502952 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.503318 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.503324 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.503328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.503479 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.503942 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.503949 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.504123 (+ 174us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.504326 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.504658 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"mutex_wait_us":25,"thread_start_us":132,"threads_started":1}
I20260924 20:24:36.505726 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.493775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60384 (local address 127.18.172.124:44983)
0924 20:24:36.494066 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.494075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.494085 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:24:36.494111 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.494113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.494155 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.494250 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.494762 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.495875 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.496959 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.497429 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497635 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.500068 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.500074 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.500084 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.500106 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.502458 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.502977 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.502983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.502985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.503182 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.503485 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.503488 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.503490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.504016 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.504147 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.505375 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.505513 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":176,"server-negotiator.queue_time_us":230,"spinlock_wait_cycles":1408,"thread_start_us":99,"threads_started":1}
I20260924 20:24:36.506114 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.491185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.18.172.124:44983)
0924 20:24:36.491736 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.491753 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.491790 (+ 37us) server_negotiation.cc:407] Connection header received
0924 20:24:36.492542 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.492583 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.493023 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.493386 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.494685 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.495754 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.496712 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.497429 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.497638 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.501298 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.501302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.501304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.501328 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.503851 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.504414 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.504419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.504420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.504501 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.504815 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.504818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.504820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.504987 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.505139 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.505823 (+ 684us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.505934 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":187,"server-negotiator.queue_time_us":362,"spinlock_wait_cycles":4352,"thread_start_us":164,"threads_started":1}
I20260924 20:24:36.507819 9185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.510708 9035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: Bootstrap starting.
I20260924 20:24:36.510694 9110 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } attempt: 3
I20260924 20:24:36.513646 9035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: Neither blocks nor log segments found. Creating new log.
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:36.514642 9035 log.cc:824] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: Log is configured to *not* fsync() on all Append() calls
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
I20260924 20:24:36.517204 9035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: No bootstrap required, opened a new log
I20260924 20:24:36.519454 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.508968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36208 (local address 127.18.172.125:46231)
0924 20:24:36.509129 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.509133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.509151 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:24:36.509188 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.509190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.509225 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.509320 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.509710 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.510201 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.511171 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.511496 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.511565 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.514311 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.514315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.514318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.514342 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.517041 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.517540 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.517543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.517545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.517626 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.518213 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.518217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.518218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.518714 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.518782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.519174 (+ 392us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.519291 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 20:24:36.519454 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.508443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.18.172.126:39299)
0924 20:24:36.508604 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.508610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.508628 (+ 18us) server_negotiation.cc:407] Connection header received
0924 20:24:36.508666 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.508669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.508711 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.508785 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.509606 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.510316 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.511359 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.511666 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.511738 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.515611 (+ 3873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.515616 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.515621 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.515642 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.517317 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.517881 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.517885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.517889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.517993 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.518349 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.518353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.518355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.518588 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.518758 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.519174 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.519291 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260924 20:24:36.519943 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.508671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:46231 (local address 127.0.0.1:36208)
0924 20:24:36.508758 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.508772 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.508843 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.509333 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.509336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.509505 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.509589 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.509594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.510311 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.510316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.511047 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.511057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.511991 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.511998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.512103 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.512796 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.512808 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.514166 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.517160 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.517166 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.517169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.517423 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.517744 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.517750 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.517754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.518103 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.518860 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.518863 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.518954 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.519634 (+ 680us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.519771 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 20:24:36.520798 9185 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:36.522050 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.511055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60386)
0924 20:24:36.511272 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.511284 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.511345 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.511869 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.511872 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.512060 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.512146 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.512155 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.513325 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.513330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.514167 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.514176 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.514794 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.514799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.514961 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.515506 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.515523 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.517423 (+ 1900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.519390 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.519394 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.519396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.519664 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.521050 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.521053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.521055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.521152 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.521603 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.521607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.521699 (+ 92us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.521775 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.521907 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260924 20:24:36.521903 9035 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.522137 9035 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff1a2385169e4aba90c7d5883e2cdf41, State: Initialized, Role: FOLLOWER
I20260924 20:24:36.522513 9035 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.522791 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.511475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.18.172.124:44983)
0924 20:24:36.511693 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.511695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.511705 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:24:36.511730 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.511731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.511759 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.511861 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.512258 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.513172 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.514290 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.514656 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.515207 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.517559 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.517562 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.517565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.517589 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.519260 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.520817 (+ 1557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.520822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.520824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.520917 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.521272 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.521275 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.521277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.521473 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.522409 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.522499 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.522617 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260924 20:24:36.524528 9035 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:36.525951 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: Bootstrap starting.
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281476, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:24:36 dist-test-slave-m1qq krb5kdc[8954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281475, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
W20260924 20:24:36.530712 9203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:36.530848 9203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:36.531736 9189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.521268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:44983 (local address 127.0.0.1:60392)
0924 20:24:36.521610 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:36.521624 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:36.521711 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:36.521989 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:36.521991 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:36.522166 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:36.522271 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.522280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.523321 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.523324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.524186 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:36.524196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.524822 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.524825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:36.524959 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.525654 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:36.525670 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:36.527877 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:36.530187 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.530191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.530195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.530437 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.530700 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:36.530702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:36.530704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:36.530781 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:36.531232 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:36.531234 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:36.531287 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 20:24:36.531408 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 20:24:36.531559 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301}
I20260924 20:24:36.531759 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.521458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60392 (local address 127.18.172.124:44983)
0924 20:24:36.521553 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.521558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.521663 (+ 105us) server_negotiation.cc:407] Connection header received
0924 20:24:36.521798 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.521800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.521833 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.521874 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.522415 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.523214 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.524298 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.524632 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.524700 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.528025 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.528029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.528032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.528055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.530065 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.530535 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.530538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.530540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.530610 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.530877 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.530881 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.530882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.531130 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.531191 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.531451 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.531615 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":37}
I20260924 20:24:36.532760 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.521259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.18.172.125:46231)
0924 20:24:36.521361 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.521366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.521526 (+ 160us) server_negotiation.cc:407] Connection header received
0924 20:24:36.521704 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.521707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.521753 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.521804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.522665 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.523411 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.524472 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.524811 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.524898 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.528056 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.528060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.528062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.528086 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.530736 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.531300 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.531304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.531307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.531390 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.531708 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.531712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.531714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.531918 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.532117 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.532515 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.532608 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 20:24:36.529987 8967 master_runner.cc:186] Error getting master registration for 127.18.172.125:46231: OK, instance_id { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" instance_seqno: 1790281476259569 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:24:36.534226 8967 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:24:36.535259 9192 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.535517 9192 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.535591 9110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:36.536402 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:36.536566 9110 log.cc:824] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:36.537428 9185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: Bootstrap starting.
I20260924 20:24:36.537750 9009 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ff1a2385169e4aba90c7d5883e2cdf41"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43432
I20260924 20:24:36.537966 9009 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:24:36.538074 9009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:36.538987 9110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: No bootstrap required, opened a new log
I20260924 20:24:36.539418 9009 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.540329 9185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:36.540513 9009 leader_election.cc:288] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [CANDIDATE]: Term 1 election: Requested vote from peers 28c4e380c30b47ef96d1fc6a38f06f20 (127.18.172.125:46231), 331d5369042d45c59648c96de2bb69aa (127.18.172.124:44983)
I20260924 20:24:36.541232 9185 log.cc:824] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:36.543736 9110 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.544023 9110 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28c4e380c30b47ef96d1fc6a38f06f20, State: Initialized, Role: FOLLOWER
I20260924 20:24:36.544384 9110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.544895 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.545048 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.545819 9110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:36.545921 9084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "28c4e380c30b47ef96d1fc6a38f06f20"
I20260924 20:24:36.546118 9084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:36.547636 9084 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1a2385169e4aba90c7d5883e2cdf41 in term 1.
I20260924 20:24:36.548147 8985 leader_election.cc:302] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28c4e380c30b47ef96d1fc6a38f06f20, ff1a2385169e4aba90c7d5883e2cdf41; no voters:
I20260924 20:24:36.548532 9192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:36.548725 9192 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [term 1 LEADER]: Becoming Leader. State: Replica: ff1a2385169e4aba90c7d5883e2cdf41, State: Running, Role: LEADER
I20260924 20:24:36.549065 9192 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.550952 9192 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1a2385169e4aba90c7d5883e2cdf41. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.551108 9192 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: This master's current role is: LEADER
W20260924 20:24:36.553467 9222 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:36.553542 9222 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:36.553803 9185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: No bootstrap required, opened a new log
I20260924 20:24:36.556188 9226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:36.557194 9226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:36.555929 9042 master_runner.cc:186] Error getting master registration for 127.18.172.124:44983: OK, instance_id { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" instance_seqno: 1790281476480396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:24:36.559350 9042 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:24:36.560194 9185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.560611 9185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331d5369042d45c59648c96de2bb69aa, State: Initialized, Role: FOLLOWER
I20260924 20:24:36.561216 9185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } }
I20260924 20:24:36.563704 9159 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "331d5369042d45c59648c96de2bb69aa"
I20260924 20:24:36.563962 9159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:36.565811 9159 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff1a2385169e4aba90c7d5883e2cdf41 in term 1.
I20260924 20:24:36.566401 9185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:36.568706 9231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.568928 9231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.576896 9190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.554182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60396 (local address 127.18.172.124:44983)
0924 20:24:36.554710 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.554714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.560275 (+ 5561us) server_negotiation.cc:407] Connection header received
0924 20:24:36.562665 (+ 2390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.562668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.562719 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.562773 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.567665 (+ 4892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.568422 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.570677 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.571091 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.571179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.572949 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.572953 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.572956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.572984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.574686 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.575285 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.575288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.575292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.575388 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.575722 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.575728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.575732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.575980 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.576153 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.576546 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.576677 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449}
I20260924 20:24:36.579802 9112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.552725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.18.172.125:46231)
0924 20:24:36.552832 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.552837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.559793 (+ 6956us) server_negotiation.cc:407] Connection header received
0924 20:24:36.561585 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.561591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.561651 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.561708 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.565753 (+ 4045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.566435 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.570725 (+ 4290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.571071 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.571161 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.575969 (+ 4808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.575973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.575976 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.576004 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.577732 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.578376 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.578380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.578381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.578469 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.578804 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.578810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.578811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.579063 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.579197 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.579544 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.579656 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:24:36.582248 9159 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [term 1 FOLLOWER]: Refusing update from remote peer ff1a2385169e4aba90c7d5883e2cdf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260924 20:24:36.582295 9242 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:36.582346 9242 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:36.582963 9192 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:36.583974 9084 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [term 1 FOLLOWER]: Refusing update from remote peer ff1a2385169e4aba90c7d5883e2cdf41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:36.584494 9192 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:36.586730 9231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1a2385169e4aba90c7d5883e2cdf41. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.586863 9231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.588176 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff1a2385169e4aba90c7d5883e2cdf41. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.588308 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.591537 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:36.552024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.18.172.126:39299)
0924 20:24:36.552137 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.552142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.555951 (+ 3809us) server_negotiation.cc:407] Connection header received
0924 20:24:36.559878 (+ 3927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.559881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.559936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.559988 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.563769 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.564531 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.569524 (+ 4993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:36.569848 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:36.569924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:36.587512 (+ 17588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:36.587517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:36.587520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:36.587553 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:36.589286 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.589817 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.589821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.589823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.589911 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:36.590193 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:36.590197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:36.590199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:36.590430 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:36.590550 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:36.591238 (+ 688us) server_negotiation.cc:299] Negotiation successful
0924 20:24:36.591372 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:24:36.594007 9223 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.594141 9223 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:36.595433 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.595556 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.596177 9192 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.596293 9192 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:36.597337 9231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff1a2385169e4aba90c7d5883e2cdf41" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 39299 } } peers { permanent_uuid: "28c4e380c30b47ef96d1fc6a38f06f20" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 46231 } } peers { permanent_uuid: "331d5369042d45c59648c96de2bb69aa" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 44983 } } }
I20260924 20:24:36.597468 9231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:36.598028 9230 mvcc.cc:205] Tried to move back new op lower bound from 7332992928044331008 to 7332992927948525568. Current Snapshot: MvccSnapshot[applied={T|T < 7332992928044331008}]
I20260924 20:24:36.598204 9246 mvcc.cc:205] Tried to move back new op lower bound from 7332992928044331008 to 7332992927948525568. Current Snapshot: MvccSnapshot[applied={T|T < 7332992928044331008}]
I20260924 20:24:36.598402 9244 mvcc.cc:205] Tried to move back new op lower bound from 7332992928044331008 to 7332992927948525568. Current Snapshot: MvccSnapshot[applied={T|T < 7332992928044331008}]
I20260924 20:24:36.606195 9226 catalog_manager.cc:1347] Generated new cluster ID: d20f826e2f19461fabaa2611b4b5186a
I20260924 20:24:36.606274 9226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:36.635511 9226 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:36.637663 9226 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:36.650910 9226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff1a2385169e4aba90c7d5883e2cdf41: Generated new TSK 0
I20260924 20:24:36.651908 9226 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:24:36.683593 9038 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.680534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.18.172.126:39299)
0924 20:24:36.680661 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.680665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.682468 (+ 1803us) server_negotiation.cc:407] Connection header received
0924 20:24:36.682682 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.682686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.682735 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.682785 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.683462 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43456: BlockingRecv error: recv got EOF from 127.0.0.1:43456 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 20:24:36.684952 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.681253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.18.172.125:46231)
0924 20:24:36.681380 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.681385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.683980 (+ 2595us) server_negotiation.cc:407] Connection header received
0924 20:24:36.684199 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.684202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.684254 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.684318 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.684839 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36236: BlockingRecv error: recv got EOF from 127.0.0.1:36236 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 20:24:36.686652 9190 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.681864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60408 (local address 127.18.172.124:44983)
0924 20:24:36.681976 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.681980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.685437 (+ 3457us) server_negotiation.cc:407] Connection header received
0924 20:24:36.685832 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.685835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.685885 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.686013 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.686538 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60408: BlockingRecv error: recv got EOF from 127.0.0.1:60408 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 20:24:36.691264 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.689636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36240 (local address 127.18.172.125:46231)
0924 20:24:36.689793 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.689797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.690015 (+ 218us) server_negotiation.cc:407] Connection header received
0924 20:24:36.690220 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.690223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.690416 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.690522 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.691154 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36240: BlockingRecv error: recv got EOF from 127.0.0.1:36240 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 20:24:36.697450 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.695862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.18.172.125:46231)
0924 20:24:36.695970 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.695974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.696523 (+ 549us) server_negotiation.cc:407] Connection header received
0924 20:24:36.696717 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.696720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.696771 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.696821 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.697323 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36252: BlockingRecv error: recv got EOF from 127.0.0.1:36252 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260924 20:24:36.703316 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.701708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36260 (local address 127.18.172.125:46231)
0924 20:24:36.701814 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.701818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.702360 (+ 542us) server_negotiation.cc:407] Connection header received
0924 20:24:36.702570 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.702573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.702625 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.702674 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.703207 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36260: BlockingRecv error: recv got EOF from 127.0.0.1:36260 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 20:24:36.711233 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.709501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36276 (local address 127.18.172.125:46231)
0924 20:24:36.709647 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.709652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.710225 (+ 573us) server_negotiation.cc:407] Connection header received
0924 20:24:36.710419 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.710422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.710474 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.710541 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.711116 (+ 575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36276: BlockingRecv error: recv got EOF from 127.0.0.1:36276 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 20:24:36.723356 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.721564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36284 (local address 127.18.172.125:46231)
0924 20:24:36.721690 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.721695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.722279 (+ 584us) server_negotiation.cc:407] Connection header received
0924 20:24:36.722498 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.722501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.722558 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.722611 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.723230 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36284: BlockingRecv error: recv got EOF from 127.0.0.1:36284 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 20:24:36.744153 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.742052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36294 (local address 127.18.172.125:46231)
0924 20:24:36.742193 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.742198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.742896 (+ 698us) server_negotiation.cc:407] Connection header received
0924 20:24:36.743180 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.743184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.743248 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.743305 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.743985 (+ 680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36294: BlockingRecv error: recv got EOF from 127.0.0.1:36294 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 20:24:36.780350 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.778899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36310 (local address 127.18.172.125:46231)
0924 20:24:36.779054 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.779059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.779221 (+ 162us) server_negotiation.cc:407] Connection header received
0924 20:24:36.779547 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.779551 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.779611 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.779709 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.780194 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36310: BlockingRecv error: recv got EOF from 127.0.0.1:36310 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260924 20:24:36.847970 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.846747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36314 (local address 127.18.172.125:46231)
0924 20:24:36.846890 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.846894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.847150 (+ 256us) server_negotiation.cc:407] Connection header received
0924 20:24:36.847237 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.847239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.847290 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.847372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.847847 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36314: BlockingRecv error: recv got EOF from 127.0.0.1:36314 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 20:24:36.979741 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:36.978364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36328 (local address 127.18.172.125:46231)
0924 20:24:36.978550 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:36.978555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:36.978718 (+ 163us) server_negotiation.cc:407] Connection header received
0924 20:24:36.978919 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:36.978922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:36.978970 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:36.979089 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:36.979587 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36328: BlockingRecv error: recv got EOF from 127.0.0.1:36328 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260924 20:24:37.239780 9112 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:37.238290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.18.172.125:46231)
0924 20:24:37.238477 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.238481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.238498 (+ 17us) server_negotiation.cc:407] Connection header received
0924 20:24:37.238726 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.238729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.238874 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.238975 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:37.239614 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36340: BlockingRecv error: recv got EOF from 127.0.0.1:36340 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 20:24:37.555402 9222 catalog_manager.cc:1259] Loaded cluster ID: d20f826e2f19461fabaa2611b4b5186a
I20260924 20:24:37.555480 9222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: loading cluster ID for follower catalog manager: success
I20260924 20:24:37.557189 9222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: acquiring CA information for follower catalog manager: success
I20260924 20:24:37.558403 9222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28c4e380c30b47ef96d1fc6a38f06f20: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:24:37.583758 9242 catalog_manager.cc:1259] Loaded cluster ID: d20f826e2f19461fabaa2611b4b5186a
I20260924 20:24:37.583871 9242 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: loading cluster ID for follower catalog manager: success
I20260924 20:24:37.585737 9242 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: acquiring CA information for follower catalog manager: success
I20260924 20:24:37.586994 9242 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 331d5369042d45c59648c96de2bb69aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:24:37.758226 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:37.754019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.18.172.125:46231)
0924 20:24:37.754271 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.754275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.754472 (+ 197us) server_negotiation.cc:407] Connection header received
0924 20:24:37.754538 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.754540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.754671 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.754776 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:37.755049 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.755646 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.756565 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.756789 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.756870 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:37.757287 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:37.757744 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:37.757847 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:37.757971 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 20:24:37.758067 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":76,"threads_started":1}
I20260924 20:24:37.765802 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:37.761605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60418 (local address 127.18.172.124:44983)
0924 20:24:37.761912 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.761916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.761979 (+ 63us) server_negotiation.cc:407] Connection header received
0924 20:24:37.762118 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.762121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.762240 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.762467 (+ 227us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:37.762686 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.763347 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.764257 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.764540 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.764629 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:37.764922 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:37.765357 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:37.765476 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:37.765539 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 20:24:37.765589 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":75,"threads_started":1}
I20260924 20:24:37.773919 9315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:37.769915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36344 (local address 127.18.172.125:46231)
0924 20:24:37.770067 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.770071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.770186 (+ 115us) server_negotiation.cc:407] Connection header received
0924 20:24:37.770402 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.770405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.770464 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.770513 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:37.770977 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.771722 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.772622 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.772890 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.772986 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:37.773326 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:37.773413 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:37.773460 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:37.773670 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 20:24:37.773738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 20:24:37.774058 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:37.770290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.18.172.124:44983)
0924 20:24:37.770404 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.770409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.770419 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:24:37.770461 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.770463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.770513 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.770583 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:37.770894 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.771473 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.772951 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.773187 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.773306 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:37.773595 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:37.773670 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:37.773715 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:37.773875 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 20:24:37.773936 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 20:24:37.774704 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:37.769360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.18.172.126:39299)
0924 20:24:37.769643 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:37.769649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:37.769671 (+ 22us) server_negotiation.cc:407] Connection header received
0924 20:24:37.769894 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:37.769897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:37.770070 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:37.770488 (+ 418us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:37.770621 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.771488 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.772939 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:37.773278 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:37.773406 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:37.773702 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:37.774084 (+ 382us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:37.774454 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:37.774490 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 20:24:37.774546 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260924 20:24:37.778173 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 8967
I20260924 20:24:37.784890 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 9042
I20260924 20:24:37.792073 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 9117
2026-09-24T20:24:37Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3217 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 20:24:37 dist-test-slave-m1qq krb5kdc[9332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 20:24:37 dist-test-slave-m1qq krb5kdc[9332](info): set up 2 sockets
Sep 24 20:24:37 dist-test-slave-m1qq krb5kdc[9332](info): commencing operation
krb5kdc: starting...
W20260924 20:24:38.912789 19121 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/test-user.keytab.
Sep 24 20:24:38 dist-test-slave-m1qq krb5kdc[9332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T20:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T20:24:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.18.172.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
Entry for principal kudu/127.18.172.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab.
I20260924 20:24:39.047155 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43107
--webserver_interface=127.18.172.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 20:24:39.159224 9345 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:39.159545 9345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:39.159569 9345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:39.159610 9345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:39.163584 9345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:39.163641 9345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:39.163661 9345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:39.163677 9345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:39.163692 9345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:39.168946 9345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.126.keytab
--principal=kudu/127.18.172.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.126:43107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.18.172.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:39.169994 9345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:39.171025 9345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:39.180038 9351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:39.180095 9345 server_base.cc:1034] running on GCE node
W20260924 20:24:39.180049 9353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:39.180051 9350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:39.180971 9345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:39.182067 9345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:39.183233 9345 hybrid_clock.cc:648] HybridClock initialized: now 1790281479183197 us; error 47 us; skew 500 ppm
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:39.190794 9345 init.cc:377] Logged in from keytab as kudu/127.18.172.126@KRBTEST.COM (short username kudu)
I20260924 20:24:39.192003 9345 webserver.cc:460] Webserver started at http://127.18.172.126:41383/ using document root <none> and password file <none>
I20260924 20:24:39.192654 9345 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:39.192770 9345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:39.193038 9345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:39.195485 9345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data/instance:
uuid: "cdee8d6ae06a4873acb90de123137805"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "8ce3733d4c4f992280afd545a19052f1"
server_key_iv: "9aff7ba8ab95e7fe6198179cd7f5f7da"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.196156 9345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance:
uuid: "cdee8d6ae06a4873acb90de123137805"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "8ce3733d4c4f992280afd545a19052f1"
server_key_iv: "9aff7ba8ab95e7fe6198179cd7f5f7da"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.200377 9345 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:24:39.203138 9360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:39.204528 9345 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 20:24:39.204778 9345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal
uuid: "cdee8d6ae06a4873acb90de123137805"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "8ce3733d4c4f992280afd545a19052f1"
server_key_iv: "9aff7ba8ab95e7fe6198179cd7f5f7da"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.204912 9345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:39.224171 9345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:39.227794 9345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:39.228032 9345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:39.235349 9345 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.126:43107
I20260924 20:24:39.235736 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.126:43107 every 8 connection(s)
I20260924 20:24:39.236361 9345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/data/info.pb
I20260924 20:24:39.239311 9413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:39.243211 9413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.244267 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 9345
I20260924 20:24:39.244397 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-0/wal/instance
I20260924 20:24:39.244696 19121 external_mini_cluster.cc:1428] Setting key a6c959176665b308aa85ff6f8bba78db
WARNING: no policy specified for kudu/127.18.172.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.125@KRBTEST.COM" created.
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:39.263079 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.245769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43107 (local address 127.0.0.1:50150)
0924 20:24:39.246556 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.246622 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.247791 (+ 1169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.248703 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.248708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.249095 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.249433 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.249458 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.251403 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.251407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.252202 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.252213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.253230 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.253234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.253328 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.254000 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.254017 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.257760 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.259874 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.259882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.259894 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.260134 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.260483 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.260486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.260488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.260605 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.261242 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.261248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.261406 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.261582 (+ 176us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.261968 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":109,"spinlock_wait_cycles":59264,"thread_start_us":150,"threads_started":1}
I20260924 20:24:39.263087 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.245769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50150 (local address 127.18.172.126:43107)
0924 20:24:39.246368 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.246385 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.246756 (+ 371us) server_negotiation.cc:407] Connection header received
0924 20:24:39.248174 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.248183 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.248456 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.248661 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.249627 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.251266 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.252337 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.253031 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.253221 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.257947 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.257954 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.257965 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.257987 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.259711 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.260256 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.260262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.260263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.260371 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.260711 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.260713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.260715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.261117 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.261262 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.261655 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.261920 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":144,"threads_started":1}
I20260924 20:24:39.265408 9413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.266207 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.265872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50488)
0924 20:24:39.266043 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.266134 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 20:24:39.266450 9364 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
I20260924 20:24:39.267480 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.267212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50492)
0924 20:24:39.267316 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.267386 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:24:39.267689 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:45373: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
Entry for principal kudu/127.18.172.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab.
I20260924 20:24:39.275672 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:45373
--webserver_interface=127.18.172.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:24:39.316977 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } attempt: 1
I20260924 20:24:39.317919 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.317583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50502)
0924 20:24:39.317769 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.317834 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 20:24:39.318795 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.318580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50512)
0924 20:24:39.318691 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.318741 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 20:24:39.318948 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:45373: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
I20260924 20:24:39.387171 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } attempt: 2
I20260924 20:24:39.388134 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.387794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50514)
0924 20:24:39.387968 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.388039 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 20:24:39.388940 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.388703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50522)
0924 20:24:39.388826 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.388879 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 20:24:39.389088 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.125:45373: Network error: Client connection negotiation failed: client connection to 127.18.172.125:45373: connect: Connection refused (error 111)
W20260924 20:24:39.392830 9420 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:39.393141 9420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:39.393191 9420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:39.393296 9420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:39.397727 9420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:39.397796 9420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:39.397817 9420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:39.397835 9420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:39.397882 9420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:39.403592 9420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.125.keytab
--principal=kudu/127.18.172.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.125:45373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.18.172.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:39.404708 9420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:39.405742 9420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:39.414444 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:39.414422 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:39.414422 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:39.414718 9420 server_base.cc:1034] running on GCE node
I20260924 20:24:39.415390 9420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:39.416551 9420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:39.417742 9420 hybrid_clock.cc:648] HybridClock initialized: now 1790281479417723 us; error 30 us; skew 500 ppm
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:39.420547 9420 init.cc:377] Logged in from keytab as kudu/127.18.172.125@KRBTEST.COM (short username kudu)
I20260924 20:24:39.421377 9420 webserver.cc:460] Webserver started at http://127.18.172.125:41427/ using document root <none> and password file <none>
I20260924 20:24:39.421939 9420 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:39.422003 9420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:39.422164 9420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:39.424171 9420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data/instance:
uuid: "447033fcce404c6b9d4dfba8f821c847"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "3fe44681dd0209b2b9485a0c4e606e6a"
server_key_iv: "4ff7b375bc56eef59c78988e22c9cd55"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.424695 9420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal/instance:
uuid: "447033fcce404c6b9d4dfba8f821c847"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "3fe44681dd0209b2b9485a0c4e606e6a"
server_key_iv: "4ff7b375bc56eef59c78988e22c9cd55"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.428794 9420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 20:24:39.431363 9435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:39.432686 9420 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260924 20:24:39.432825 9420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal
uuid: "447033fcce404c6b9d4dfba8f821c847"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "3fe44681dd0209b2b9485a0c4e606e6a"
server_key_iv: "4ff7b375bc56eef59c78988e22c9cd55"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.432940 9420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:39.449669 9420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:39.453042 9420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:39.453270 9420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:39.461272 9420 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.125:45373
I20260924 20:24:39.461655 9487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.125:45373 every 8 connection(s)
I20260924 20:24:39.462350 9420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/data/info.pb
I20260924 20:24:39.462709 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 9420
I20260924 20:24:39.462849 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-1/wal/instance
I20260924 20:24:39.463217 19121 external_mini_cluster.cc:1428] Setting key 15ce6cabf728239893627026644a4440
I20260924 20:24:39.465900 9488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:39.469849 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.18.172.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.18.172.124@KRBTEST.COM" created.
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:39.480259 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } attempt: 3
I20260924 20:24:39.485903 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.471879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.18.172.126:43107)
0924 20:24:39.472018 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.472022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.473113 (+ 1091us) server_negotiation.cc:407] Connection header received
0924 20:24:39.474131 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.474134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.474196 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.474302 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.475614 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.476149 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.477308 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.477539 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.477655 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.480613 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.480616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.480619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.480644 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.482608 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.483232 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.483236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.483238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.483339 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.483748 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.483752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.483754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.484033 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.484162 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.485551 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.485686 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 20:24:39.486425 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.472269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43107 (local address 127.0.0.1:50160)
0924 20:24:39.472959 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.473002 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.473916 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.474462 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.474469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.474910 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.475458 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.475480 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.476281 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.476285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.477163 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.477172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.477697 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.477702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.477881 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.478513 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.478529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.480409 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.482719 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.482725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.482740 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.483122 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.483447 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.483450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.483452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.483621 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.484188 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.484195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.484479 (+ 284us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.484742 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.485159 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":176,"threads_started":1}
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:39.488221 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.491964 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.480782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50538)
0924 20:24:39.481416 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.481437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.481542 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.482265 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.482270 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.482621 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.482781 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.482789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.484071 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.484075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.484694 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.484709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.485354 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.485358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.485475 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.485961 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.485980 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.487602 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.489893 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.489898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.489902 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.490196 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.490554 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.490556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.490557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.490726 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.491376 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.491379 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.491481 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.491635 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.491782 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540}
I20260924 20:24:39.492065 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.481022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50538 (local address 127.18.172.125:45373)
0924 20:24:39.481400 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.481414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.481810 (+ 396us) server_negotiation.cc:407] Connection header received
0924 20:24:39.481866 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.481874 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.482102 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.482251 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.482924 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.483954 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.484829 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.485186 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.485348 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.487733 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.487741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.487774 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.487805 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.489770 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.490329 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.490335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.490336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.490431 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.490885 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.490889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.490890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.491238 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.491404 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.491732 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.491883 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:39.494437 9413 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.495344 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.495054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47308)
0924 20:24:39.495202 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.495262 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
Entry for principal kudu/127.18.172.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab.
I20260924 20:24:39.496256 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.495946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47316)
0924 20:24:39.496137 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.496195 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 20:24:39.496891 19121 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQK3utR/build/debug/bin/kudu
/tmp/dist-test-taskQK3utR/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:39805
--webserver_interface=127.18.172.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.18.172.84:32935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 20:24:39.498183 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.488992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50542)
0924 20:24:39.489186 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.489202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.489471 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.489843 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.489848 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.490038 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.490164 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.490172 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.491286 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.491292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.491962 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.491971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.492599 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.492603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.492745 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.493199 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.493210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.494461 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.496678 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.496681 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.496682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.496917 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.497237 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.497239 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.497240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.497315 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.497776 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.497778 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.497890 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.497974 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.498074 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":25}
I20260924 20:24:39.498467 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.489036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50542 (local address 127.18.172.125:45373)
0924 20:24:39.489368 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.489380 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.489539 (+ 159us) server_negotiation.cc:407] Connection header received
0924 20:24:39.489588 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.489593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.489646 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.489751 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.490321 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.491178 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.492115 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.492489 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.492601 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.494613 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.494617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.494619 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.494644 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.496554 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.497028 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.497035 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.497039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.497133 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.497411 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.497417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.497422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.497677 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.497783 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.498058 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.498197 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":251,"thread_start_us":113,"threads_started":1}
W20260924 20:24:39.498526 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.499197 9488 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.499903 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.499640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47322)
0924 20:24:39.499755 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.499838 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 20:24:39.500165 9438 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.501016 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.500812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47330)
0924 20:24:39.500931 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.500972 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 20:24:39.501235 9488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.529704 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 1
I20260924 20:24:39.530637 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.530268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47342)
0924 20:24:39.530453 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.530535 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 20:24:39.531436 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.531203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47354)
0924 20:24:39.531315 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.531367 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 20:24:39.531589 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.550513 9488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 1
I20260924 20:24:39.551497 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.551128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47362)
0924 20:24:39.551307 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.551400 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 20:24:39.552307 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.552031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47374)
0924 20:24:39.552167 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.552221 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 20:24:39.552481 9488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.606801 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 2
I20260924 20:24:39.607815 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.607380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47384)
0924 20:24:39.607673 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.607734 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260924 20:24:39.608600 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.608347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47392)
0924 20:24:39.608466 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.608517 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 20:24:39.608757 9413 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
W20260924 20:24:39.611446 9495 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 20:24:39.611745 9495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 20:24:39.611799 9495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 20:24:39.611886 9495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 20:24:39.616055 9495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 20:24:39.616107 9495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 20:24:39.616129 9495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 20:24:39.616146 9495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 20:24:39.616163 9495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 20:24:39.620666 9488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 2
I20260924 20:24:39.621462 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.621179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47398)
0924 20:24:39.621321 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.621388 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 20:24:39.621917 9495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.18.172.84:32935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.18.172.126:43107,127.18.172.125:45373,127.18.172.124:39805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/krb5kdc/kudu_127.18.172.124.keytab
--principal=kudu/127.18.172.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.18.172.124:39805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.18.172.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 24 Sep 2026 19:20:07 UTC on ccycloud.quasar-jcuhlt.root.comops.site
build id 4670859
I20260924 20:24:39.622237 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.621957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47404)
0924 20:24:39.622095 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.622173 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 20:24:39.622397 9488 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.18.172.124:39805: Network error: Client connection negotiation failed: client connection to 127.18.172.124:39805: connect: Connection refused (error 111)
I20260924 20:24:39.623153 9495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 20:24:39.624183 9495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 20:24:39.632855 9501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:39.632843 9502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 20:24:39.633062 9504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 20:24:39.633242 9495 server_base.cc:1034] running on GCE node
I20260924 20:24:39.633824 9495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 20:24:39.634951 9495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 20:24:39.636176 9495 hybrid_clock.cc:648] HybridClock initialized: now 1790281479636161 us; error 27 us; skew 500 ppm
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 20:24:39.639146 9495 init.cc:377] Logged in from keytab as kudu/127.18.172.124@KRBTEST.COM (short username kudu)
I20260924 20:24:39.640106 9495 webserver.cc:460] Webserver started at http://127.18.172.124:44625/ using document root <none> and password file <none>
I20260924 20:24:39.640669 9495 fs_manager.cc:359] Metadata directory not provided
I20260924 20:24:39.640729 9495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 20:24:39.640962 9495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 20:24:39.643064 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data/instance:
uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "e1e7c029594b8fea86ab5f93a1a0c072"
server_key_iv: "209519eef588913bbf53e7cd3137172d"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.643694 9495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal/instance:
uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "e1e7c029594b8fea86ab5f93a1a0c072"
server_key_iv: "209519eef588913bbf53e7cd3137172d"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.647796 9495 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 20:24:39.650956 9511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 20:24:39.652289 9495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 20:24:39.652405 9495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data,/tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal
uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7"
format_stamp: "Formatted at 2026-09-24 20:24:39 on dist-test-slave-m1qq"
server_key: "e1e7c029594b8fea86ab5f93a1a0c072"
server_key_iv: "209519eef588913bbf53e7cd3137172d"
server_key_version: "encryptionkey@0"
I20260924 20:24:39.652565 9495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 20:24:39.687336 9495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 20:24:39.690382 9495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 20:24:39.690594 9495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 20:24:39.697736 9495 rpc_server.cc:307] RPC server started. Bound to: 127.18.172.124:39805
I20260924 20:24:39.698038 9563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.18.172.124:39805 every 8 connection(s)
I20260924 20:24:39.698752 9495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/data/info.pb
I20260924 20:24:39.701581 9564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 20:24:39.704792 19121 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQK3utR/build/debug/bin/kudu as pid 9495
I20260924 20:24:39.704931 19121 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQK3utR/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790281188136637-19121-0/minicluster-data/master-2/wal/instance
I20260924 20:24:39.705215 19121 external_mini_cluster.cc:1428] Setting key cbcdea037361a5c0ac8175b98b8aea58
I20260924 20:24:39.705313 9564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.707929 9413 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 3
I20260924 20:24:39.713563 9488 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } attempt: 3
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.126@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.125@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
I20260924 20:24:39.721848 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.708309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47414)
0924 20:24:39.708462 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.708475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.708580 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.709182 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.709186 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.709362 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.709480 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.709489 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.711281 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.711285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.711951 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.711960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.712638 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.712641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.712726 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.713201 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.713215 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.715311 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.719676 (+ 4365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.719680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.719682 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.719948 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.720305 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.720308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.720309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.720394 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.721309 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.721313 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.721416 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.721527 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.721662 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 20:24:39.722283 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.708371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.18.172.124:39805)
0924 20:24:39.708720 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.708727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.708737 (+ 10us) server_negotiation.cc:407] Connection header received
0924 20:24:39.708764 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.708766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.709007 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.709633 (+ 626us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.709758 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.711153 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.712097 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.712512 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.712634 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.715462 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.715471 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.715486 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.715509 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.719551 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.720091 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.720100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.720105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.720192 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.720516 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.720519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.720520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.720937 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.721294 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.721627 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.721786 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":305,"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":18432,"thread_start_us":147,"threads_started":1}
I20260924 20:24:39.722445 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.706698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47410 (local address 127.18.172.124:39805)
0924 20:24:39.707145 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.707160 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.707192 (+ 32us) server_negotiation.cc:407] Connection header received
0924 20:24:39.708397 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.708420 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.709023 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.709182 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.710086 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.711153 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.713758 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.714157 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.714365 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.717135 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.717139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.717142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.717170 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.719676 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.720147 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.720153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.720156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.720216 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.720516 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.720519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.720520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.720937 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.721167 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.722179 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.722286 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":25728,"thread_start_us":116,"threads_started":1}
I20260924 20:24:39.721850 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.707338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.126:43107 (local address 127.0.0.1:50168)
0924 20:24:39.707756 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.707810 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.708486 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.708959 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.708969 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.709450 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.709839 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.709851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.710637 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.710642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.711607 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.711617 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.712119 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.712123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.712313 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.713074 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.713099 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.715463 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.717930 (+ 2467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.717941 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.717953 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.718384 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.718713 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.718716 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.718718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.718936 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.719585 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.719592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.719938 (+ 346us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.720159 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.720507 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260924 20:24:39.723377 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.707473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.18.172.126:43107)
0924 20:24:39.707604 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.707608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.707911 (+ 303us) server_negotiation.cc:407] Connection header received
0924 20:24:39.708729 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.708732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.708785 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.708870 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.709976 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.710526 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.711726 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.711966 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.712080 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.715662 (+ 3582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.715667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.715672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.715704 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.717797 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.718510 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.718514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.718517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.718606 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.719111 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.719117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.719120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.719432 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.719529 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.722246 (+ 2717us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.723213 (+ 967us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 20:24:39.724666 9491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.714013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47418)
0924 20:24:39.715079 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.715095 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.715183 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.715504 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.715508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.715705 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.715833 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.715841 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.717097 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.717102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.718015 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.718030 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.718710 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.718713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.718804 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.719257 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.719271 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.720622 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.723034 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.723038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.723040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.723284 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.723624 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.723629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.723634 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.723729 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.724159 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.724162 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.724261 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.724352 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.724481 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950}
I20260924 20:24:39.724984 9564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } has no permanent_uuid. Determining permanent_uuid...
I20260924 20:24:39.725159 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.714056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.18.172.124:39805)
0924 20:24:39.714342 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.714352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.715270 (+ 918us) server_negotiation.cc:407] Connection header received
0924 20:24:39.715307 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.715310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.715356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.715956 (+ 600us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.716076 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.716965 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.718168 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.718558 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.718696 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.720764 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.720767 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.720769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.720792 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.722877 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.723409 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.723415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.723420 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.723524 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.723841 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.723844 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.723846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.724046 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.724166 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.724877 (+ 711us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.724980 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":2944,"thread_start_us":105,"threads_started":1}
I20260924 20:24:39.726817 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: Bootstrap starting.
I20260924 20:24:39.728318 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: Bootstrap starting.
I20260924 20:24:39.729140 9413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:39.730014 9413 log.cc:824] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:39.730839 9488 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: Neither blocks nor log segments found. Creating new log.
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.126@KRBTEST.COM
I20260924 20:24:39.731710 9488 log.cc:824] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:39.732146 9413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: No bootstrap required, opened a new log
I20260924 20:24:39.734628 9488 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: No bootstrap required, opened a new log
I20260924 20:24:39.736195 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.725377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.18.172.126:43107)
0924 20:24:39.725484 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.725488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.726049 (+ 561us) server_negotiation.cc:407] Connection header received
0924 20:24:39.726086 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.726088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.726134 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.726201 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.727029 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.727753 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.728807 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.729108 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.729204 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.732285 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.732290 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.732296 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.732326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.733952 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.734439 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.734443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.734444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.734544 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.734854 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.734858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.734859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.735102 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.735262 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.735928 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.736038 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
I20260924 20:24:39.736501 9413 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.736724 9413 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdee8d6ae06a4873acb90de123137805, State: Initialized, Role: FOLLOWER
I20260924 20:24:39.737247 9413 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.738862 9413 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:39.739379 9488 raft_consensus.cc:348] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.739580 9488 raft_consensus.cc:729] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447033fcce404c6b9d4dfba8f821c847, State: Initialized, Role: FOLLOWER
I20260924 20:24:39.739873 9488 consensus_queue.cc:260] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.741181 9488 sys_catalog.cc:564] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:39.741425 9573 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.742308 9573 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.742257 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.725852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.18.172.125:45373)
0924 20:24:39.725962 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.725966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.731164 (+ 5198us) server_negotiation.cc:407] Connection header received
0924 20:24:39.731331 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.731334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.731384 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.731452 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.732136 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.732946 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.734284 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.734644 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.734723 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.737717 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.737721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.737723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.737747 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.739533 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.740275 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.740281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.740284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.740367 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.740729 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.740735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.740736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.741030 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.741111 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.741681 (+ 570us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.741804 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":48}
I20260924 20:24:39.742930 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.725637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.125:45373 (local address 127.0.0.1:50546)
0924 20:24:39.731087 (+ 5450us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.731105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.731231 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.731596 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.731599 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.731900 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.732004 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.732011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.733319 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.733324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.734142 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.734153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.734923 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.734927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.735060 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.735535 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.735549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.737568 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.739783 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.739787 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.739789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.740149 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.740502 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.740505 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.740507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.740594 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.741193 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.741196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.741379 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.742575 (+ 1196us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.742728 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 20:24:39.743902 9564 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.18.172.125@KRBTEST.COM
W20260924 20:24:39.748898 9585 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:39.748980 9585 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:39.747853 9420 master_runner.cc:186] Error getting master registration for 127.18.172.124:39805: OK, instance_id { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" instance_seqno: 1790281479695750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:24:39.751199 9420 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 20:24:39.751375 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.737640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50556 (local address 127.18.172.125:45373)
0924 20:24:39.737749 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.737753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.737902 (+ 149us) server_negotiation.cc:407] Connection header received
0924 20:24:39.738062 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.738065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.738112 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.738159 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.739057 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.739798 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.740856 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.741184 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.741273 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.747299 (+ 6026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.747303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.747305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.747334 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.749133 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.749742 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.749746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.749748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.749845 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.750177 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.750180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.750182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.750400 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.750591 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.750960 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.751171 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:24:39.747215 9345 master_runner.cc:186] Error getting master registration for 127.18.172.124:39805: OK, instance_id { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" instance_seqno: 1790281479695750 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 20:24:39.751516 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.751633 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.751641 9345 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 24 20:24:39 dist-test-slave-m1qq krb5kdc[9332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790281479, etypes {rep=17 tkt=17 ses=17}, kudu/127.18.172.124@KRBTEST.COM for kudu/127.18.172.124@KRBTEST.COM
I20260924 20:24:39.754747 19121 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 20:24:39.756044 9387 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "cdee8d6ae06a4873acb90de123137805"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50178
I20260924 20:24:39.756304 9387 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 20:24:39.756424 9387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:39.757856 9387 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
W20260924 20:24:39.758611 9596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:39.758697 9596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:39.758869 9387 leader_election.cc:288] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [CANDIDATE]: Term 1 election: Requested vote from peers 447033fcce404c6b9d4dfba8f821c847 (127.18.172.125:45373), 43992c6bcd9a4c179bd77dfa9fa3a5d7 (127.18.172.124:39805)
I20260924 20:24:39.759723 9462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdee8d6ae06a4873acb90de123137805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "447033fcce404c6b9d4dfba8f821c847"
I20260924 20:24:39.759846 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.744579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.18.172.124:39805)
0924 20:24:39.744695 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.744699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.746454 (+ 1755us) server_negotiation.cc:407] Connection header received
0924 20:24:39.746497 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.746499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.746544 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.746616 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.750169 (+ 3553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.751022 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.752086 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.752396 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.752485 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.754893 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.754897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.754899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.754924 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.756699 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.757178 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.757182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.757183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.757262 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.758838 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.758841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.758843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.759111 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.759193 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.759536 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.759662 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 20:24:39.759964 9462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:39.760345 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.744337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.18.172.124:39805 (local address 127.0.0.1:47422)
0924 20:24:39.745835 (+ 1498us) negotiation.cc:107] Waiting for socket to connect
0924 20:24:39.745849 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 20:24:39.745933 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 20:24:39.749764 (+ 3831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 20:24:39.749768 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 20:24:39.749945 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 20:24:39.750053 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.750059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.751139 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.751142 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.751967 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 20:24:39.751978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.752621 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.752624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 20:24:39.752698 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.753210 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 20:24:39.753225 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0924 20:24:39.754750 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 20:24:39.756818 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.756821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.756824 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.757069 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.757360 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 20:24:39.757363 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 20:24:39.757365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 20:24:39.757451 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 20:24:39.759249 (+ 1798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 20:24:39.759251 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 20:24:39.759346 (+ 95us) client_negotiation.cc:769] Sending connection context
0924 20:24:39.760029 (+ 683us) client_negotiation.cc:240] Negotiation successful
0924 20:24:39.760167 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1428}
I20260924 20:24:39.761817 9462 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdee8d6ae06a4873acb90de123137805 in term 1.
I20260924 20:24:39.762642 9364 leader_election.cc:302] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447033fcce404c6b9d4dfba8f821c847, cdee8d6ae06a4873acb90de123137805; no voters:
I20260924 20:24:39.762949 9573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 20:24:39.763162 9573 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [term 1 LEADER]: Becoming Leader. State: Replica: cdee8d6ae06a4873acb90de123137805, State: Running, Role: LEADER
I20260924 20:24:39.763464 9564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: Bootstrap starting.
I20260924 20:24:39.763484 9573 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.765767 9573 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdee8d6ae06a4873acb90de123137805. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.765885 9573 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:39.766304 9564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: Neither blocks nor log segments found. Creating new log.
I20260924 20:24:39.767288 9564 log.cc:824] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: Log is configured to *not* fsync() on all Append() calls
I20260924 20:24:39.771667 9608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 20:24:39.777081 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.762185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50562 (local address 127.18.172.125:45373)
0924 20:24:39.762306 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.762310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.764935 (+ 2625us) server_negotiation.cc:407] Connection header received
0924 20:24:39.765103 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.765108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.765154 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.765204 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.768209 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.768976 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.771109 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.771475 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.771565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.773450 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.773454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.773457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.773484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.775288 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.775765 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.775769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.775771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.775848 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.776121 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.776124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.776126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.776325 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.776453 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.776788 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.776915 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:24:39.779419 9608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 20:24:39.782773 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.761486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.18.172.126:43107)
0924 20:24:39.761675 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.761680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.763502 (+ 1822us) server_negotiation.cc:407] Connection header received
0924 20:24:39.763701 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.763704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.763753 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.763804 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.766138 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.766945 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.770142 (+ 3197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.770489 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.770580 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.778368 (+ 7788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.778372 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.778375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.778413 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.780451 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.781149 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.781153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.781154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.781240 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.781669 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.781672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.781674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.781885 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.782004 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.782495 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.782624 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260924 20:24:39.785032 9564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: No bootstrap required, opened a new log
I20260924 20:24:39.789922 9564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.790217 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:39.762828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47424 (local address 127.18.172.124:39805)
0924 20:24:39.762930 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.762934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.774617 (+ 11683us) server_negotiation.cc:407] Connection header received
0924 20:24:39.775500 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.775504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.775564 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.775619 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.782767 (+ 7148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.783557 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.784576 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:39.784920 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:39.785023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:39.786612 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 20:24:39.786616 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 20:24:39.786619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 20:24:39.786642 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 20:24:39.788318 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.788845 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.788848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.788850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.788938 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 20:24:39.789238 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 20:24:39.789241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 20:24:39.789243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 20:24:39.789446 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 20:24:39.789604 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:39.789834 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 20:24:39.789988 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 20:24:39.790238 9564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43992c6bcd9a4c179bd77dfa9fa3a5d7, State: Initialized, Role: FOLLOWER
I20260924 20:24:39.790588 9564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } }
I20260924 20:24:39.791234 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.791383 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.796176 9564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 20:24:39.796221 9538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cdee8d6ae06a4873acb90de123137805" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7"
I20260924 20:24:39.796464 9538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 20:24:39.797992 9538 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cdee8d6ae06a4873acb90de123137805 in term 1.
I20260924 20:24:39.802330 9462 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [term 1 FOLLOWER]: Refusing update from remote peer cdee8d6ae06a4873acb90de123137805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:39.802577 9538 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [term 1 FOLLOWER]: Refusing update from remote peer cdee8d6ae06a4873acb90de123137805: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 20:24:39.802949 9605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 20:24:39.803450 9605 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 20:24:39.806005 9622 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 20:24:39.806124 9622 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 20:24:39.807333 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdee8d6ae06a4873acb90de123137805. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.807471 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.809538 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.809666 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:39.810923 9608 catalog_manager.cc:1347] Generated new cluster ID: 867a14e1ea8242c3b23f4caf9ee4adf5
I20260924 20:24:39.810981 9608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 20:24:39.815349 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdee8d6ae06a4873acb90de123137805. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.815460 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.815601 9611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.815689 9611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.823592 9624 mvcc.cc:205] Tried to move back new op lower bound from 7332992941260800000 to 7332992941116350464. Current Snapshot: MvccSnapshot[applied={T|T < 7332992941260800000}]
I20260924 20:24:39.824755 9608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 20:24:39.825923 9608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 20:24:39.826001 9605 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.826107 9605 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805 [sys.catalog]: This master's current role is: LEADER
I20260924 20:24:39.828614 9579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cdee8d6ae06a4873acb90de123137805" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cdee8d6ae06a4873acb90de123137805" member_type: VOTER last_known_addr { host: "127.18.172.126" port: 43107 } } peers { permanent_uuid: "447033fcce404c6b9d4dfba8f821c847" member_type: VOTER last_known_addr { host: "127.18.172.125" port: 45373 } } peers { permanent_uuid: "43992c6bcd9a4c179bd77dfa9fa3a5d7" member_type: VOTER last_known_addr { host: "127.18.172.124" port: 39805 } } }
I20260924 20:24:39.828740 9579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 20:24:39.829471 9627 mvcc.cc:205] Tried to move back new op lower bound from 7332992941260800000 to 7332992941116350464. Current Snapshot: MvccSnapshot[applied={T|T < 7332992941260800000}]
I20260924 20:24:39.840126 9608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdee8d6ae06a4873acb90de123137805: Generated new TSK 0
I20260924 20:24:39.841006 9608 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 20:24:39.873720 9416 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.870405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50188 (local address 127.18.172.126:43107)
0924 20:24:39.870534 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.870539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.872406 (+ 1867us) server_negotiation.cc:407] Connection header received
0924 20:24:39.872816 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.872820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.872874 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.872929 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.873580 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50188: BlockingRecv error: recv got EOF from 127.0.0.1:50188 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 20:24:39.875483 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.871137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.18.172.125:45373)
0924 20:24:39.871255 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.871259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.872729 (+ 1470us) server_negotiation.cc:407] Connection header received
0924 20:24:39.874308 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.874311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.874368 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.874420 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.875358 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50570: BlockingRecv error: recv got EOF from 127.0.0.1:50570 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260924 20:24:39.876569 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.871794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.18.172.124:39805)
0924 20:24:39.871904 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.871908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.874225 (+ 2317us) server_negotiation.cc:407] Connection header received
0924 20:24:39.875830 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.875834 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.875884 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.875933 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.876451 (+ 518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47438: BlockingRecv error: recv got EOF from 127.0.0.1:47438 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260924 20:24:39.881204 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.879735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50578 (local address 127.18.172.125:45373)
0924 20:24:39.879832 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.879836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.880310 (+ 474us) server_negotiation.cc:407] Connection header received
0924 20:24:39.880518 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.880521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.880593 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.880643 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.881113 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50578: BlockingRecv error: recv got EOF from 127.0.0.1:50578 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 20:24:39.885911 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.884532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.18.172.125:45373)
0924 20:24:39.884627 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.884631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.885121 (+ 490us) server_negotiation.cc:407] Connection header received
0924 20:24:39.885288 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.885291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.885335 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.885382 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.885822 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50592: BlockingRecv error: recv got EOF from 127.0.0.1:50592 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 20:24:39.891242 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.889786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.18.172.125:45373)
0924 20:24:39.889875 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.889879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.890362 (+ 483us) server_negotiation.cc:407] Connection header received
0924 20:24:39.890534 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.890537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.890582 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.890629 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.891124 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50608: BlockingRecv error: recv got EOF from 127.0.0.1:50608 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 20:24:39.898819 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.897166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.18.172.125:45373)
0924 20:24:39.897305 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.897309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.897871 (+ 562us) server_negotiation.cc:407] Connection header received
0924 20:24:39.898081 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.898085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.898136 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.898188 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.898720 (+ 532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50624: BlockingRecv error: recv got EOF from 127.0.0.1:50624 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 20:24:39.911155 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.909278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.18.172.125:45373)
0924 20:24:39.909407 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.909412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.910084 (+ 672us) server_negotiation.cc:407] Connection header received
0924 20:24:39.910293 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.910298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.910366 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.910431 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.910989 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50630: BlockingRecv error: recv got EOF from 127.0.0.1:50630 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 20:24:39.931559 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.929661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50646 (local address 127.18.172.125:45373)
0924 20:24:39.929801 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.929806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.930437 (+ 631us) server_negotiation.cc:407] Connection header received
0924 20:24:39.930695 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.930699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.930756 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.930812 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.931438 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50646: BlockingRecv error: recv got EOF from 127.0.0.1:50646 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260924 20:24:39.967276 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:39.965911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50656 (local address 127.18.172.125:45373)
0924 20:24:39.966078 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:39.966083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:39.966324 (+ 241us) server_negotiation.cc:407] Connection header received
0924 20:24:39.966504 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:39.966507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:39.966558 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:39.966655 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:39.967149 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50656: BlockingRecv error: recv got EOF from 127.0.0.1:50656 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260924 20:24:40.035243 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:40.033866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50668 (local address 127.18.172.125:45373)
0924 20:24:40.034035 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.034041 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.034299 (+ 258us) server_negotiation.cc:407] Connection header received
0924 20:24:40.034443 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.034446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.034497 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.034581 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:40.035119 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50668: BlockingRecv error: recv got EOF from 127.0.0.1:50668 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 20:24:40.166762 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:40.165516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50672 (local address 127.18.172.125:45373)
0924 20:24:40.165678 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.165683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.165860 (+ 177us) server_negotiation.cc:407] Connection header received
0924 20:24:40.166050 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.166054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.166102 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.166195 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:40.166656 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50672: BlockingRecv error: recv got EOF from 127.0.0.1:50672 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260924 20:24:40.426713 9493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 20:24:40.425454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.18.172.125:45373)
0924 20:24:40.425596 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.425600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.425712 (+ 112us) server_negotiation.cc:407] Connection header received
0924 20:24:40.425896 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.425898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.425947 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.426009 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 20:24:40.426572 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50688: BlockingRecv error: recv got EOF from 127.0.0.1:50688 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 20:24:40.760030 9596 catalog_manager.cc:1259] Loaded cluster ID: 867a14e1ea8242c3b23f4caf9ee4adf5
I20260924 20:24:40.760116 9596 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: loading cluster ID for follower catalog manager: success
I20260924 20:24:40.761875 9596 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: acquiring CA information for follower catalog manager: success
I20260924 20:24:40.763137 9596 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 447033fcce404c6b9d4dfba8f821c847: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:24:40.807703 9622 catalog_manager.cc:1259] Loaded cluster ID: 867a14e1ea8242c3b23f4caf9ee4adf5
I20260924 20:24:40.807791 9622 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: loading cluster ID for follower catalog manager: success
I20260924 20:24:40.809687 9622 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: acquiring CA information for follower catalog manager: success
I20260924 20:24:40.810791 9622 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 43992c6bcd9a4c179bd77dfa9fa3a5d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 20:24:40.945453 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:40.940948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.18.172.125:45373)
0924 20:24:40.941203 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.941207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.941335 (+ 128us) server_negotiation.cc:407] Connection header received
0924 20:24:40.941659 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.941662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.941798 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.941934 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:40.942157 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.942884 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.943941 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.944233 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.944322 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:40.944639 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:40.945107 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:40.945201 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:40.945274 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 20:24:40.945324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":95,"threads_started":1}
I20260924 20:24:40.952628 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:40.948765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.18.172.124:39805)
0924 20:24:40.949028 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.949031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.949044 (+ 13us) server_negotiation.cc:407] Connection header received
0924 20:24:40.949123 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.949125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.949238 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.949335 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:40.949571 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.950089 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.950970 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.951278 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.951368 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:40.951699 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:40.952157 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:40.952253 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:40.952371 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 20:24:40.952425 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":79,"threads_started":1}
I20260924 20:24:40.960664 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:40.956536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.18.172.125:45373)
0924 20:24:40.956668 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.956672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.956787 (+ 115us) server_negotiation.cc:407] Connection header received
0924 20:24:40.956987 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.956992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.957043 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.957093 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:40.957451 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.958249 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959441 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.959825 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959931 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:40.960188 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:40.960260 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:40.960343 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:40.960454 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 20:24:40.960520 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 20:24:40.960750 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:40.956026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47448 (local address 127.18.172.124:39805)
0924 20:24:40.956141 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.956145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.956761 (+ 616us) server_negotiation.cc:407] Connection header received
0924 20:24:40.956949 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.956952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.957000 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.957047 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:40.957458 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.958244 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959363 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.959714 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959810 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:40.960220 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:40.960286 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:40.960385 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:40.960524 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 20:24:40.960585 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 20:24:40.960883 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 20:24:40.955941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50202 (local address 127.18.172.126:43107)
0924 20:24:40.956170 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 20:24:40.956177 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 20:24:40.956513 (+ 336us) server_negotiation.cc:407] Connection header received
0924 20:24:40.956644 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 20:24:40.956647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 20:24:40.956822 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 20:24:40.956937 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 20:24:40.957272 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.957900 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959058 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 20:24:40.959451 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 20:24:40.959551 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 20:24:40.959900 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 20:24:40.960274 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 20:24:40.960369 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 20:24:40.960666 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 20:24:40.960732 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":59,"threads_started":1}
I20260924 20:24:40.964316 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 9345
I20260924 20:24:40.971609 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 9420
I20260924 20:24:40.977640 19121 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQK3utR/build/debug/bin/kudu with pid 9495
2026-09-24T20:24:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3185 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6402 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (292834 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST